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2.  Do  you  believe  this  research  topic  is  significant  enough  that  it  would 
have  been  researched  (or  contracted)  by  your  organization  or  another  agency 
if  AFIT  had  not  researched  it? 


3.  The  benefits  of  AFIT  research  can  often  be  expressed  by  the  equivalent 
value  that  your  agency  received  by  virtue  of  AFIT  performing  the  research. 

Can  you  estimate  what  this  research  would  have  cost  if  it  had  been 
accomplished  under  contract  or  if  it  had  been  done  in-house  in  terms  of  man¬ 
power  and/or  dollars? 

a.  Man-years _  $ _ (Contract) . 

b.  Man-years _  $ _ (In-house). 

4.  Often  it  is  not  possible  to  attach  equivalent  dollar  values  to  research, 
although  die  results  of  the  research  may,  in  fact,  be  important.  Whether  or 
not  you  were  able  to  establish  an  equivalent  value  for  this  research  (3  above) 
what  is  your  estimate  of  its  significance? 

a.  Highly  b.  Significant  c.  Slightly  d.  Of  No 
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CHAPTER  I 


INTRODUCTION 


Overview 

The  declining  defense  budget  dictates  a  need  to  explore  ways  to 
cut  the  ownership  costs  associated  with  any  weapon  system.  Owner¬ 
ship  costs  could  be  defined  as  acquisition  costs  and  operating  and 
support  (OfeS)  costs.  Recently,  increased  emphasis  has  been  placed 
upon  reducing  the  OfcS  costs  of  a  system  in  addition  to  acquisition 
costs .  This  is  in  recognition  that  O&S  costs  are  the  largest  portion 
of  total  ownership  costs.  One  way  of  reducing  O&S  costs  is  to  make 
the  systems  more  reliable. 

The  services  examined  the  commercial  airlines  industry 
because  of  the  airlines'  past  ability  to  consistently  acquire  more 
reliable  equipment  than  the  services  (8:B-2  to  B-5).  One  distin¬ 
guishing  characteristic  of  their  acquisition  methodology  was  the  use 
of  warranties .  The  Navy  became  the  first  service  to  apply  the 
lessons  learned  from  the  airlines  with  a  warranty  concept  that  would 
come  to  be  called  the  Reliability  Improvement  Warranty  (RIW).  RTW 
i®  ®  fixed  price  commitment  that  involves  contractor  repair  or 
replacement  of  equipment  that  fails  during  the  period  of  warranty 
coverage  (4:A-4  and  A-5).  Apparent  success  of  those  early  warranty 
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effort*  led  to  the  current  trial  RIW  application  program. 


Since  the  RIW  wa*  a  relatively  new  contracting  technique,  it 
followed  that  detailed  information  as  well  as  policy  guidance  was 
needed  by  personnel  in  the  field.  This  information  was  disseminated 
to  Air  Force  personnel  in  July,  1974,  via  a  pamphlet  entitled.  Interim 


Guidelines  Reliability  Improvement  Warranty  (RIW).  published  by  the 
HQ  USAF  Directorate  of  Procurement  Policy  (LGP).  These  guide- 
lines  are  still  used  as  an  aid  in  evaluating  the  potential  benefits  to  be 
gained  from  the  application  of  RIW  In  a  contract  and  as  general 
guidance  on  constructing  a  RIW  contract.  An  evaluation  of  the  ade¬ 
quacy  of  the  present  Air  Foifce  RIW  Guidelines  is  the  subject  of  this 
research. 

Problem  Statement 

Based  upon  the  increasing  popularity  of  RIW  and  its  potential 
for  misuse,  the  increasing  DOD  involvement  in  international  copro¬ 
duction  programs  which  could  involve  RIW,  and  lessons  learned  from 
previous  RIW  applications,  there  is  a  need  to  evaluate  the  current 
RIW  guidelines  and  recommend  changes  thereto. 

Background 

Weapon  system  reliability,  or  the  lack  there  of,  is  a  problem 
that  persistently  plagues  managers  at  all  levels,  not  only  in  the  United 
States  Air  Force  but  across  the  entire  Department  of  Defense.  One 


need  not  reflect  very  long  to  recall  instances  where  the  lack  of  reli- 
ability  in  a  weapon  system  affected  him  personally.  Such  instances 
are,  at  best,  inconvenient.  Frequently  the  consequences  are  much 
worse  in  terms  of  unsuccessful  missions,  increased  maintenance 
costs,  and  even  lives  lost. 

What  is  meant  by  the  term  "reliability"?  There  are  any  num¬ 
ber  of  definitions,  many  of  which  seem  to  define  reliability  in  terms 
of  itself.  One  of  the  better  definitions  is  in  Military  Standard  (MIL- 
STD)  721B,  Definitions  of  Effectiveness  Terms  for  Reliability. 


Maintainability.  Human  Factors,  and  Safety  which  defines  reliability 
as  "  .  .  .  the  probability  that  a  system  will  perform  satisfactorily 
for  a  given  period  of  time  when  used  under  stated  conditions  [34:7]." 

Because  of  the  very  visible  effect  on  maintainability  and  logis¬ 
tics  support  costs,  the  tendency,  in  the  Air  Force,  has  been  to  relate 
the  reliability  management  task  to  the  logisticians,  i.e.,  to  the 
people  who  support,  rather  than  buy  or  fly,  systems.  Therefore, 
reliability  is  most  often  expressed  in  terms  of  mean  time  between 
failure  (MTBF)  or  mean  time  between  maintenance  action  (MTBMA). 
Logistics  reliability  requirements  are  frequently  traded  off  against 
traditional  performance  goals  (speed,  accuracy,  altitude,  endurance, 
etc.)  when  determining  the  allocation  of  scarce  resources,  such  as 
time  and  money.  The  overall  goal  to  be  sought  during  these  trade¬ 
offs  is  availability.  According  to  General  Bryce  Poe,  AFLC 


commander,  "Availability  ia  the  key  [39]"  to  the  success  of  the  Air 
Force  mission.  Since  reliability  is  the  overwhelmingly  important 
factor  in  availability,  especially  as  weapon  systems  age  and  become 
more  prone  to  failure,  more  reliability  is  urgently  needed.  General 
Poe  illustrated  his  thinking  with  this  example.  If  an  engineer  says  he 
can  design  an  aircraft  to  takeoff  on  a  shorter  runway,  or  fly  higher  or 
faster  than  specified,  we  need  to  tell  him  to  turn  down  the  wick- -give 
us  the  "specs"  and,  make  it  last  longer  (39).  Therefore,  in  as  much 
as  reliability  directly  affects  availability,  it  also  affects  other  issues, 
such  as  strike  force  size  and  total  force  size.  Reliability  should  be 
considered  as  much  a  "performance"  factor  as  speed  or  endurance 
(43:15-16). 

Increasing  costs.  Coupled  with  the  very  real  need  for  reliable  equip¬ 
ment,  the  declining  defense  budget  makes  it  imperative  to  explore 
ways  to  cut  the  ownership  costs  associated  with  any  weapon  system. 
The  cost  of  a  weapon  system  is  very  similar  to  an  iceberg.  Research 
and  development  as  well  as  procurement  costs  are  the  visible  portion, 
but  the  costs  to  operate  and  maintain  the  system  are  less  visible. 
Former  AFLC  Commander  General  Rogers  commented: 

Until  recently,  government  and  industrial  design  engineers 
were  motivated  primarily  by  performance,  schedule,  and  cost 
considerations  during  this  process.  But  operating  and  support 
outlays  have  gradually  surpassed  the  historically  higher  acqui¬ 
sition  costs  and  grown  out  of  proportion  to  the  original  costs  of 
some  weapon  systems  [42:47]. 
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Because  the  cost  problem  is  across  the  complete  life  cycle,  the  entire 
weapon  system  life  cycle  must  be  addressed  in  order  to  solve  the  cost 
problem  (1:1-36  to  1-37). 

There  are  three  acquisition  methods  currently  in  use  which  are 
intended  to  deal  with  the  very  real  weapon  system  ownership  costs. 
Life  cycle  costing  considers  life  cycle  cost  in  the  various  decisions 
associated  with  acquiring  an  item  of  equipment  or  a  weapon  system. 
Design  to  cost  is  a  management  concept  to  control  acquisition  and 
support  costs  through  quantified  cost  goals.  The  third  method,  reli¬ 
ability  improvement  warranty  (RTW),  will  be  discussed  in  some  depth 
in  the  following  paragraphs  (1:1-38  to  1-43). 

Early  warranty  applications .  The  success  of  the  commercial  air 
transport  industry's  procurement  process  with  the  warranty  has  not 
gone  unnoticed  by  the  DOD.  This  process  has  proven  effective  in 
providing  the  airlines  highly  reliable  equipment  at  competitive  prices. 
In  comparing  Air  Force  and  commercial  airline  instrument  landing 
systems,  believed  to  be  comparable  in  performance,  operational 
characteristics,  and  age,  the  commercial  systems  were  acquired  for 
approximately  half  the  price  of  the  Air  Force  systems,  but  the  MTBF 
of  the  commercial  systems  was  roughly  two  to  four  times  that  of  the 
Air  Force  systems.  One  of  the  predominant  features  of  the  airline 
procurement  environment  is  the  extensive  use  of  warranty  (1:8-25  to 


8-28). 

In  1972,  delivery  was  completed  on  126  additional  A24G-27 
gyros  for  the  F/FB-111  aircraft.  These  gyros  were  procured  through 
competition  and  the  contract  included  a  long-term  warranty  provision 
calling  for  3,000  hours  or  five  years  of  use.  The  acquisition  cost  of 
these  126  gyros  was  37%  lower  than  the  gyros  initially  bought  and  the 
MTBF  was  35%  longer  by  mid- 1974  (1:9-7  to  9-8). 

To  prove  that  their  hydraulic  pump  had  inherent  design  features 
which  would  result  in  higher  efficiencies  and  longer  life  than  any  com¬ 
petitor's  pump,  Abex  Corporation  very  aggressively  pursued  any 
procurement  methodology  during  the  late  1960s  which  utilized  high 
reliability  and/or  lower  life  cycle  cost  as  criteria  for  award.  Thus 
the  Abex  model  AP27V-5-02  pump  was  installed  on  the  Navy's  F-14A 
fighter  as  contractor  furnished  equipment.  By  mid- 1975  the  mean 
time  between  return  of  these  pumps  had  increased  to  more  than  800 
operating  hours  which  exceeded  the  specified  end  point  of  750  oper¬ 
ating  hours  per  return  (1:9-9  to  9-10). 

Reliability  Improvement  Warranty 

Out  of  this  backdrop  of  warranty  application  the  Reliability 
Improvement  Warranty  concept  was  developed. 

A  Reliability  Improvement  Warranty  is  defined  as  a  provision 
in  either  a  fixed  price  acquisition,  or  fixed  price  equipment 
overhaul  contract  in  which: 

(a)  the  contractor  is  provided  with  a  monetary  Incentive, 
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throughout  the  period  of  the  warranty,  to  improve  the  production 
design  and  engineering  of  the  equipment  so  as  to  enhance  the 
field/operational  reliability  and  maintainability  of  the  system/ 
equipment;  and 

(b)  the  contractor  agrees  that  during  a  specified  or  measured 
period  of  use,  he  will  repair  or  replace  (within  a  specified  turn¬ 
around  time)  all  equipment  that  fails  (subject  to  specified  exclu¬ 
sions  if  applicable)  [4:A-4  and  A-5]. 

It  should  be  noted  that  the  reliability  improvement  warranty  is 
not  simply  a  maintenance  contract.  The  contractor  is  not  required 
to  perform  routine  preventive  maintenance,  cleaning,  etc.  Addition¬ 
ally,  the  RIW  does  not  cover  items  such  as  filters  and  light  bulbs  that 
are  expected  to  need  replacement  in  normal  operation  (4:A-1  to  A-2). 

The  objective  of  RIW  is 

...  to  motivate  and  provide  an  incentive  to  contractors  to 
design  and  produce  equipment  which  will  have  a  low  failure  rate 
as  well  as  low  repair  costs,  after  failure  due  to  field/operational 
use.  Furthermore  this  technique  attempts  ...  to  provide  an 
incentive  for  contractors  to  improve  the  reliability  of  their 
equipment  and  to  reduce  repair  costs  during  the  period  of  warranty 
coverage  in  order  to  maximize  their  profits  [4:A-l]. 


Incentives .  The  idea  of  incentives  as  it  relates  to  RIW  was  pointed 
out  very  vividly  in  an  ARINC  Research  Corporation  report. 

The  incentive  power  of  RIW  can  best  be  examined  by  com¬ 
paring  the  normal  procurement  with  a  warranty  procurement. 

In  a  normal  procurement  a  contractor  may  realize  his  maximum 
profit  delivering  the  lowest  reliability  that  will  be  accepted  by 
the  customer.  Not  only  does  he  reduce  his  manufacturing  and 
quality- control  cost,  but  he  enhances  his  opportunity  for  sale 
of  spare  units,  service  contracts,  and  replacement  parts  to 
provide  necessary  operational  support.  .  .  .  Under  RIW,  the 
contractor  takes  on  an  obligation  for  maintenance  support  for 
a  specified  period  on  a  fixed-price  basis;  thus  his  profitability 
is  tied  to  field  reliability.  Ideally,  then,  he  will  be  motivated 
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to  increase  reliability  to  reduce  his  costs  [6:2-2]. 

Refer  to  Figures  1  and  2  for  a  graphical  presentation  of  these  con¬ 
cepts. 

The  firm  fixed  price  contract,  generally  established  by  com¬ 
petitive  bidding,  provides  this  incentive.  Competition  can,  therefore, 
be  considered  one  of  the  keys  to  the  application  of  the  RIW  contracting 
technique  (29).  In  one  sense,  a  contractor  is  in  competition  with  the 
Air  Force's  maintenance  complex,  because  RIW  will  not  be  utilized 
unless  there  is  an  expected  cost/benefit  advantage  to  doing  so. 

The  reliability  improvement  warranty  is  frequently  used  in  con¬ 
junction  with  a  mean  time  between  failure  guarantee  which  requires 
the  contractor  to  institute  corrective  action  and  to  provide  consign¬ 
ment  spares  in  the  event  the  guaranteed  level  is  not  met.  A  relia¬ 
bility  improvement  warranty  contract  is  designed  to  motivate  the 
contractor  to  improve  MTBF  through  his  maintenance  support  com¬ 
mitment.  The  MTBF  guarantee  is  an  even  stronger  incentive  because 
of  the  contractor's  obligation  to  provide  spares  to  relieve  pipeline 
shortages  (6:vii). 

Carrying  the  idea  of  contractor  incentive  a  little  further,  there 
is  reason  to  believe  that  the  contractor's  active  involvement  in  the 
operational  phase  of  a  system's  life  cycle  will  have  the  added  benefit 
of  improving  equipment's  resistance  to  system  induced  failures,  such 
as  voltage  fluctuations .  Realistically  speaking,  such  failures  are  of 
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|  Minimum  Acceptable  Reliability- 


Contractor  '» 
Initial  Cost 


Contract  Price  • 
Without  RIW 


EFFECT  OF  RELIABILITY  ON  INITIAL  COST  (26: Fig.  1) 

Figure  1 


EFFECT  OF  RELIABILITY  ON  INITIAL  AND  SUPPORT  COSTS 

(26:Flg.  2) 

Figure  2 


little  concern  to  a  contractor  under  a  conventional  contract,  but  if  the 
contractor  becomes  concerned,  unproved  circuit  designs  or  com¬ 
ponents  can  be  the  result  (4:xiii).  This  idea  was  summed  up  very 
well  by  Gandara  and  Rich. 

The  RTW  is  distinctive  as  a  "reliability  technique"  in  that  it 
involves  the  contractor  during  the  support  phase  of  his  product's 
life,  seeks  to  shift  at  least  some  of  the  reliability  risk  to  him, 
and  is  introduced  in  the  transaction  early  enough  to  allow  respon¬ 
sive  design  and  engineering  improvements  [17:30]. 


Expected  benefits.  There  are  a  number  of  benefits  which  the  govern¬ 
ment  is  expected  to  receive  through  the  RTW  contracting  procedure 
(4: A- 16  and  A- 17): 

a.  The  contractor  assumes  the  responsibility  for  field  reli¬ 
ability. 

b.  Life  cycle  costs  are  emphasized  to  a  greater  extent. 

c.  Keeping  all  units  in  the  same  configuration  is  also  the  con¬ 
tractor's  responsibility. 

d.  The  contractor  will  be  more  motivated  to  introduce  changes 
that  will  increase  the  equipment's  MTBF,  resulting  in  reliability 
growth. 

e.  There  is  an  incentive  for  the  contractor  to  reduce  repair 
costs;  such  reductions  will  increase  his  profit  or  minimize  his  loss. 

f.  Initial  support  investment  required  by  the  Government  will 
be  minimal. 


10 


g.  The  requirement  for  skilled  military  maintenance  and 
support  manpower  may  be  reduced  through  RTW  usage. 

Another  source  pointed  out  an  additional  benefit  to  the  customer 
(Government).  The  using  organization  will  be  better  able  to  budget 
future  operating  costs,  because  any  required  maintenance  is  per¬ 
formed  by  the  contractor  as  part  of  the  fixed  price  contract  (25:55). 

It  would  certainly  be  unreasonable  to  think  contractors  would 
enter  into  any  arrangement  that  would  only  benefit  the  Government. 
Expected  benefits  to  contractors  include: 

a.  When  the  MTBF  is  improved  above  the  pricing  base,  th?re 
is  an  increased  profit  potential. 

b.  There  is  multi-year  guaranteed  business. 

c.  Becoming  more  familiar  with  the  operational  and  maintain¬ 
ability  characteristics  of  his  equipment  should  help  the  contractor  in 
obtaining  follow-on  contracts  (4:A-17).  The  fact  that  all  failed  units 
are  returned  to  the  factory  for  maintenance  allows  feedback  to  occur 
much  more  rapidly  than  otherwise  possible.  Engineers  can  learn  of 
design  or  component  weaknesses  quicker  and  can  take  corrective 
action  (25:55). 

V 

Risk  shifting.  Risk  shifting  was  introduced  earlier  and  is  an  idea 
that  should  be  discussed  in  more  detail.  The  idea  that  contractors 
are  being  asked  to  assume  totally  the  risks  associated  with  system 


field  problems  has  caused  some  misgivings.  It  must  be  understood 
that  the  Government  is  paying  a  fixed  price  for  a  RIW  that  is  expected 
to  cover  all  maintenance  costs.  Any  risk  on  the  contractor's  part 
results  from  the  actual  cost  exceeding  the  warranty  price.  There¬ 
fore,  he  is  able  to  reduce  his  risk  to  a  point  of  his  choosing, 
considering  the  competitive  pressure  of  the  procurement  and  the 
Government's  willingness  to  take  the  warranty  option.  In  the  same 
manner,  the  Government  risks  paying  too  much,  if  the  system  per¬ 
forms  better  than  expected  (6:xiv). 

Negative  aspects .  A  number  of  the  positive  aspects  of  the  RIW  have 
been  pointed  out,  but  there  are  also  certain  potential  disadvantages 
(6:3-2): 

a.  The  potential  for  legal  disputes  is  high.  Therefore,  close 
coordination  between  the  user,  logistic,  and  legal  activities  is 
required  until  experience  is  gained  in  RIW  contracting  techniques. 

b.  System  acquisition  costs  maybe  increased  as  the  result  of 
the  contractor's  expending  additional  effort  to  achieve  good  initial 
reliability,  and  spares  cost  may  be  increased  as  a  result  of  sparing 
at  the  line  replaceable  unit  level  rather  than  at  the  module  level. 

c.  An  increase  in  the  number  of  military  maintenance  techni¬ 
cians  is  sure  to  be  required,  when  the  transition  to  organic  main¬ 
tenance  occurs.  Thus,  one  of  the  expected  benefits  quickly  becomes 
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a  problem  in  reobtaining  the  skilled  people. 


d.  The  ability  to  respond  quickly  in  emergency  situations  may 
be  impaired,  thus  new  or  modified  procedures  must  be  developed  to 
support  the  RIW  concept  without  adversely  affecting  the  logistic 
management  function. 

RIW  Guidelines 

The  Assistant  Secretary  of  Defense  (Installations  and  Logistics) 
directed,  in  August,  1973,  trial  use  of  warranties  by  the  services 
".  .  .in  the  acquisition  and  initial  operational  support  of  a  number  of 
Electronic  Subsystems  [5:1]."  Responding  to  the  DOD  directive,  the 
HQ  USAF  Directorate  of  Procurement  Policy  (LGP)  published  a 
pamphlet  entitled  Interim  Guidelines  Reliability  Improvement  War¬ 
ranty  (RIW)  in  July,  1974  (51).  This  pamphlet  is  referenced  in  both 
the  AFLC  and  AFSC  supplements  to  the  Defense  Acquisition  Regula¬ 
tion,  paragraph  1-324.50,  as  a  guide  to  be  used  in  the  application  of 
RTW  (50).  It  provides  guidance  in  the  areas  of  application  criteria, 
funding,  essential  contract  clause  elements,  and  determination  of  cost 
effectiveness  (51:ii).  Henceforth,  this  pamphlet  will  be  referred  to 
as  the  Air  Force  RTW  guidelines. 

The  Air  Force  RIW  guidelines  are  intended  to  aid  in  determining 
whether  economic  and  reliability  benefits  can  be  achieved  through  the 
use  of  the  RIW .  A  summary  of  the  contents  of  the  guidelines  is 
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contained  in  Table  1.  The  following  paragraphs  discuss  some  of  the 
more  important  aspects  of  the  guidelines. 

Generally  speaking,  guidelines  are  not  inflexible,  and  this  is 
the  case  with  the  Air  Force  guidelines.  There  are,  for  example, 
fourteen  application  criteria  in  these  guidelines.  Two  of  the  criteria 
are  "The  equipment  has  a  potential  for  both  reliability  growth  and 
reduction  in  repair  costs"  and  "There  is  a  reasonable  degree  of 
assurance  there  will  be  a  high  utilization  of  the  equipment  [51:10,  11]." 
The  flexibility  of  the  Air  Force  RIW  guidelines  is  illustrated  by  the 
foitbwing  extract  (51:11). 

It  is  noted  that  the  equipment  need  not  meet  all  the  criteria 
shown  above  in  order  to  apply  an  RIW.  Rather,  at  this  point  in 
time,  we  should  pick  logical  candidates  meeting  several  or  many 
of  the  criteria  so  that  further  assessment  can  be  made  of  the 
value  of  this  technique. 

Policy  guidance  contained  in  the  Air  Force  RIW  guidelines  has 
clarified  at  least  one  source  of  difficulty  in  applying  the  RIW  con¬ 
tracting  technique.  For  example,  funding  the  RIW  raises  the  question 
of  who  pays  for  the  warranty.  The  guidance  is  that  "RIWs  shall  be 
funded  from  the  same  appropriation  as  the  acquisition  or  overhaul 
warranted  [51;  14] . " 

In  August,  1974,  the  DOD  validated  the  concepts  contained  in 
the  Air  Force  RIW  guidelines  by  issuing  an  abbreviated  set  of  guide¬ 
lines  (4)  that  closely  paralled  the  Air  Force  RIW  guidelines.  The 
differences  Involved  minor  structural  changes  and  the  omission  of 
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Potential  Benefits  from  Use  of  RIWs 
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Contractor  Obligations 
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CONTENTS  OF  THE  RIW  GUIDELINES  (51) 
Table  1 


several  appendices  involving  RIW  clause  examples  and  RIW  data 
report  examples. 

Evolution  of  the  RIW  guidelines .  Both  the  DOD  and  die  Air  Force  RIW 
guidelines  evolved  from  past  experiences  and  from  managerial 
attempts  to  obtain  more  reliable  equipment  and  to  hold  costs  down. 
Rising  operating  and  maintenance  costs  clearly  underscored  the  need 
for  innovation  in  contracting  and  acquisition  techniques  in  the  DOD. 
Generally,  efforts  to  induce  contractors  to  design  reliability  and 
low-cost  support  features  were  not  successful.  Such  attempts  in¬ 
cluded  (1)  combining  target  MTBF  in  performance  specifications  with 
unit  production  cost  goals  and  (2)  soliciting  life-cycle  cost  (LCC) 
estimates  from  competing  contractors  for  incorporation  into  the 
source  selection  decision  process  (17:5). 

Commercial  airline  acquisition  methodology.  Avionics  equipment 
form,  fit,  and  function  interchangeability  profiles  had  been  developed 
since  the  mid  1930s  for  use  by  the  airlines  industry  as  part  of  the 
commercial  airlines  acquisition  methodology  (CAAM).  A  predominant 
feature  of  CAAM  was  the  use  of  warranty  clauses  in  acquisition  con¬ 
tracts  (19:2-1  thru  2-34).  The  profiles  and  contracting  techniques  of 
CAAM  were  the  forerunners  of  the  DOD  and  Air  Force  RIW  guide¬ 
lines.  A  detailed  study  of  CAAM  conducted  by  ARINC  Corporation 
resulted  in  formal  construction  of  a  list  of  CAAM  guidelines  that 


closely-  resembled  the  present  RIW  guidelines,  and  in  fact,  formed 
their  basis. 


Failure  Free  Warranty.  The  Navy's  1967  contract  for  the  repair  of 
gyroscope  platforms  was  the  first  DOD  application  of  a  reliability 
warranty.  At  that  time  the  concept  was  called  Failure  Free  War¬ 
ranty  (FFW)  (46:12  and  12A).  Three  aspects  of  the  FFW  differed 
greatly  from  past  contracting  experiences:  (1)  all  field  failures  were 
covered  under  warranty  for  an  extended  time  period,  (2)  risk  was 
shifted  so  that  the  contractor  shared  any  reward  (expanded  profit) 
or  penalty  (fixed  price)  with  the  government  and  (3)  reliability  was 
measured  in  field  operation  after  hardware  acceptance  (28:26-30). 
The  term  Reliability  Improvement  Warranty  eventually  came  to  be 
preferred  over  the  term  Failure  Free  Warranty  because  of  the  impli¬ 
cation  that  there  would  be  no  failures  during  the  warranty  period,  or 
that  the  warranty  would  be  free.  The  lessons  learned  during  the 
administration  of  this  and  other  early  applications  further  contributed 
to  the  evolving  guidelines  (45:13). 

Justification  for  Research 

The  purpose  of  the  research  is  to  determine  the  adequacy  of 
the  Air  Force  RIW  guidelines  and  to  suggest  improvements,  if  neces¬ 
sary.  The  research  is  based  upon  the  fact  that  the  RIW  is  a  trial 
concept  that  is  evolving  within  a  changing  acquisition  environment. 
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Keeping  in  step  with  change  is  difficult --the  preface  of  die  DOD  RTW 
guidelines  implies  the  need  for  research  as  a  means  to  improvement, 
as  this  portion  taken  from  the  preface  indicates: 

To  realize  the  maximum  potential  from  the  use  of  RIWs,  it  is 
important  not  only  to  identify  the  good  results  but  also  to  identify 
the  problem  areas  so  that  the  latter  can  be  factored  in  and 
corrected  in  the  Guidelines  [4:2], 

A  phase  of  this  factoring  process  was  accomplished  one  year  after 
publication  in  response  to  critical  comments  from  the  Council  of 
Defense  and  Space  Industry  Associations.  Industry  comments  indi¬ 
cated  a  perception  that  the  RIW  concept  placed  undue  risk  upon  the 
contractors  due  to  the  schedule,  and  development  constraints. 

In  April,  1976,  Headquarters  USAF  levied  extensive  review 
requirenients  upon  RIW  use.  The  requirement  for  the  reviews  was 
reiterated  in  an  AFALD  Headquarters  policy  letter  in  February,  1978, 
with  these  additional  comments: 

Perhaps  when  more  experience  with  RIWs  has  been  gained,  we 
can  suggest  a  reduction  in  emphasis  on  the  joint  reviews. 
Experience  indicates  that  better  control  and  integration  of  RIW 
provisions  is  a  valid  need  at  this  time  [44:2]. 

The  difficulties  of  control  and  integration  of  RIW  provisions  were 

restated  in  a  March,  1978,  ASD  Headquarters  policy  letter.  "Several 

recent  instances  indicate  that  some  of  our  procurement  personnel  are 

not  aware  of  the  procedures  governing  selection,  preparation,  review 

and  approval  of  RIW  provisions  [41:1]." 

Also,  in  a  recent  interview  with  AFALD  personnel,  it  was 
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learned  that  RIW  users  saw  the  effect  of  RIW  as  being  substantially 
different  from  what  was  presented  in  the  guidelines.  RIW  was  thought 
to  be  more  strongly  related  to  production  quality  than  to  reliability 
improvement  during  the  warranty  period  (16). 

Finally,  application  of  RIW  to  systems  involved  in  international 
coproduction  is  not  addressed  in  the  guidelines.  There  are  potential 
problems  in  this  area  with  increased  complexity  of  contract  terms, 
longer  lead  times,  and  political  implications  (13). 

Research  Objectives 

1.  To  determine  if  the  Air  Force  RIW  guidelines  are  adequate. 

2.  To  obtain  recommendations  for  improving  those  guidelines, 
if  necessary. 

Research  Questions 

1.  Do  the  Air  Force  personnel  involved  in  the  contracting  pro- 
cess  perceive  the  Air  Force  RIW  guidelines  as  adequate? 

2.  What  changes  should  be  made  to  the  Air  Force  RIW  guide¬ 
lines  ? 
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CHAPTER  II 


RESEARCH  METHOD 


Overview 

Chapter  I  provided  a  general  background  and  justification  for 
research  on  the  subject  of  the  RIW  guidelines.  The  research  objec¬ 
tives  were  to  determine  if  the  RIW  guidelines  are  adequate  and  to 
improve  the  guidelines,  if  necessary.  The  research  questions  indi¬ 
cated  the  method  of  achieving  the  objectives  was  to  collect  the 
perceptions  of  Air  Force  personnel,  having  RIW  relevant  experience, 
concerning  the  adequacy  of  the  guidelines.  Suggestions  on  changes 
that  should  be  made  to  the  guidelines  to  achieve  adequacy  were 
obtained  from  those  same  personnel. 

Chapter  II  specifies  the  variables  that  were  considered  in  the 
study.  This  includes  operational  definitions  of  the  variables  and  a 
breakdown  of  the  specific  attributes  of  the  variables .  Also  there  is 
a  discussion  of  the  instrument  construction  and  validation  process-- 
in  short,  how  validity  and  reliability  were  obtained  both  in  the  instru¬ 
ment,  and  in  its  application.  Then  a  discussion  of  the  data  collection 
plan  describes  the  population,  sample,  sampling  plan,  and  how  the 
data  was  gathered  under  the  attributes  of  adequacy.  This  will  be 
followed  by  the  Data  Analysis  Plan  which  (a)  explains  how  the 


collected  data  was  treated  to  bring  it  to  a  quantifiable  measure,  (b) 
establishes  the  criteria,  hypothesis  tests,  and  significance  levels 
that  were  used  to  test  each  attribute  of  adequacy  to  determine  if  the 
criteria  were  met  and  if  the  test  was  significant.  Finally  the  assump¬ 
tions  necessary  in  the  employment  of  the  research  method  and  the 
limitations  of  the  findings  derived  from  the  research  are  listed. 

Variable  Selection  and  Specification 

This  study  deals  with  the  "perceptions"  of  Air  Force  personnel 
concerning  the  adequacy  of  the  RTW  guidelines.  An  interview  research 
approach,  in  which  new  data  (respondent  perceptions)  were  created 
and  analyzed,  was  determined  to  be  appropriate.  This  section  of  the 
research  method  chapter  explains  the  selection  and  specification  of 
the  variables  and  the  construction  and  validation  of  the  interview 
guide  that  was  used  to  gather  the  research  data. 

The  research  questions  specify  what  is  to  be  measured,  i.e., 
the  adequacy  of  the  RIW  guidelines  and  possible  changes  to  improve 
them.  Therefore,  the  two  variables  in  the  study  are  adequacy  and 
need  for  change.  These  variables  are  the  mechanisms  of  measure¬ 
ment  by  which  the  research  questions  are  answered.  Specification 
of  the  variables  is  aided  through  operational  definitions.  An  opera¬ 
tional  definition  is  based  on  observable,  measurable  characteristics 
that  are  associated  with  what  is  being  defined.  A  unique  definition  is 
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desirable.  That  is,  the  more  exclusively  associated  the  observable 
criteria  and  the  defined  quantity,  the  more  useful  is  that  operational 
definition  (49:57-65). 

As  a  first  step  in  operationalizing  adequacy,  a  dictionary  defin¬ 
ition  is,  "equal  to  or  sufficient  for  some  (specific)  requirement 
[52:31]."  Further,  selected  attributes  of  adequacy  amplify  this 
definition;  two  attributes  identified  are:  complete  and  helpful.  Given 
these  attributes  the  following  operational  definition  is  provided  for 
adequate: 

1.  Complete.  The  guidelines  are  complete,  if  the  user  does  not 
have  to  refer  to  other  sources  to  determine  the  issues  to  be  con¬ 
sidered. 

2.  Helpful.  The  guidelines  are  helpful,  if  use  of  the  guidelines  aid 
the  users  in  arriving  at  decisions  concerning  RIW  issues. 

Need  for  change  can  be  operationally  defined  through  its  close 
connection  to  adequacy.  Simply  stated,  the  guidelines  need  to  be 
changed,  if  the  guidelines  are  determined  to  be  inadequate  in  either 
of  the  aforementioned  attributes. 

Instrument  Construction  and  Validation 

An  important  factor  in  instrument  construction  is  validity. 

There  are  two  main  types  of  validity- -internal  and  external.  Internal 
validity  is  the  ability  to  measure  accurately  what  is  supposed  to  be 
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measured.  On  the  other  hand,  external  validity  is  the  amount  of  con¬ 
fidence  that  can  be  placed  in  generalizations  from  the  research 
conclusions.  External  validity  is  dealt  with  in  the  discussion  of  the 
data  collection  plan.  It  is  important  to  note  that  satisfactory  external 
validity  is  impossible  without  satisfactory  internal  validity  (14:120). 
An  important  part  of  internal  validity  is  instrument  question  con¬ 
struction. 

Emory  (14:229)  summarizes  four  important  areas  in  developing 
questions  for  a  survey  instrument.  These  areas  are  (1)  question  con¬ 
tent,  (2)  question  wording,  (3)  question  response  form,  and  (4)  ques¬ 
tion  sequence. 

Each  of  these  areas  is  important  in  developing  effective  ques¬ 
tions.  For  the  first  area,  question  content,  the  main  concerns  were, 
can  and  will  the  interviewees  respond  meaningfully.  This  necessi¬ 
tates  questions  that  are  of  proper  scope,  and  appropriate  to  the 
study  effort  and  the  subject  area.  The  guide  was  developed  with  a 
specific  population  in  mind,  i.e.,  those  who  have  RTW  relevant 
experience  and  who  would  not  have  to  rely  on  second  hand  perceptions 
from  others .  By  specifying  the  population  in  this  way  all  respondents 
should  be  able  to  provide  meaningful  information.  The  questions  were 
centered  around  current  issues  in  the  RTW  area  that  would  be  familiar 
to  those  who  have  worked  with  RIW.  The  questions  were  designed  to 
be  specific  enough  to  avoid  ambiguity.  Additionally,  the  interview 
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technique  permits  clarification  where  necessary.  Finally,  the  sub¬ 
ject  area  is  not  of  a  sensitive  nature  and  willingness  to  participate 
freely  is  expected. 

Under  question  wording,  each  question  was  designed  to  be  as 
clear  as  possible,  i.e.,  stated  in  terms  of  a  shared  vocabulary  and 
without  biased  or  unnecessarily  restrictive  wording.  Difficulties  in 
this  area  were  reduced  by  a  review  of  the  interview  guide  by  selected 
persons  who  were  knowledgeable  in  the  field  but  not  part  of  the  popu¬ 
lation. 

An  oral  interview  question  format  was  chosen  for  the  study. 
This  format  was  selected,  because  it  allowed  greater  depth  of 
response  and  presented  the  best  opportunities  for  response-keyed 
questions.  The  oral  interview  allowed  recording  of  spontaneous 
elaborations  and  provided  for  more  complete  answers.  Also,  since 
people  find  it  easier,  in  general,  to  talk  than  to  write,  the  oral 
interview  was  expected  to  generate  more  information. 

In  addition  to  the  above  reasons  for  selecting  an  interview  for¬ 
mat,  another  important  reason  was  practicality.  As  Emory  notes, 
"The  scientific  requirements  of  a  project  call  for  the  measurement 
process  to  be  reliable  and  valid,  while  the  operational  requirements 
call  for  it  to  be  practical  [14:126]."  The  main  aspects  of  practicality 
that  resulted  in  the  format  chosen  were  economy  and  convenience. 
This  format  seemed  to  offer  the  best  blend  of  validity  and  practicality 
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in  collecting  the  required  data. 

The  questions  asked  during  the  interview  required  two  different 
response  modes.  A  portion  of  the  questions  on  the  interview  guide 
elicited  scaled  response,  while  another  section  contained  open  ended 
type  questions.  Both  sections  were  used  in  determining  guideline 
adequacy.  The  combination  of  open  ended  ar  t  scaled  questions  pro¬ 
vided  a  good  combination  of  response  modes  in  which  the  tendency 
toward  biasing  with  a  scaled  response  (e.g.,  the  tendency  to  rate  on 
the  middle  of  the  scale)  could  be  moderated  by  the  open  ended  response 
in  which  a  fuller  response  was  possible.  Also,  the  open  ended  ques¬ 
tions  were  intended  to  elicit  responses  that  suggested  areas  of  needed 
change  within  the  two  attributes  of  adequacy,  that  were  not  structured 
into  questions,  or  were  not  considered.  Additionally,  the  responses 
to  open  ended  questions  provided  some  ideas  for  the  changes  that 
should  be  made  to  remedy  deficiencies  that  were  brought  up  during  the 
interview. 

Attention  was  given  to  question  sequence  mainly  in  that  the 
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classification  questions  were  placed  at  the  end  of  the  interview  guide, 
in  order  to  avoid  any  reactions  of  defensiveness  that  may  occur  in 
response  to  questions  that  may  be  viewed  as  private  or  embarrassing. 
Because  the  guide  was  designed  to  be  brief,  relatively  little  attention 
was  placed  on  sequence  within  the  sections,  especially  sincefhe 
questions  were  of  about  equal  difficulty  and  covered  only  one  area  each. 
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The  attention  given  to  these  areas  of  question  construction  form 
a  limited  basis  for  instrument  validity.  This  base  was  strengthened 
through  a  review  of  the  interview  guide  by  knowledgeable  AFIT 
faculty  and  selected  base  personnel.  This  was  a  limited  review  to 
uncover  ambiguities,  or  areas  of  uncertainty.  An  independent  review 
was  selected  because  of  the  small  number  of  people  who  were  avail¬ 
able  to  form  the  sample.  It  was  anticipated  that  incorporating  the 
independent  group's  reactions  to  the  form,  content  and  usefulness  of 
the  guide  to  obtain  the  desired  information  provided  an  adequate 
amount  of  validity  for  the  purposes  of  this  study. 

Reliability,  i.e.,  consistency,  is  a  necessary  part  of  validity 
and  was  considered  in  the  instrument  design  through  the  development 
of  the  interview  guide  questions .  The  reliability  attained  was  gained 
in  part  through  definition  of  adequacy  in  terms  of  its  attributes  which 
were,  in  turn,  operationally  defined  before  being  included  in  the 
interview  guide.  Use  of  these  multiple  indicators  should  result  in 
improved  reliability,  in  that,  if  adequacy  is  indicated  throughout,  it 
will  have  been  viewed  in  two  different  ways.  Finally,  the  instrument 
was  administered  uniformly  by  both  researchers.  The  first  few 
interviews  were  conducted  by  both  researchers  to  establish  a  common 
understanding  on  the  details  of  how  the  interview  was  to  be  conducted. 
Since  no  problems  surfaced  during  this  mini  pilot  study,  these  inter¬ 
views  were  included  as  part  of  the  sample. 


Prior  to  beginning  the  interview  the  purpose  and  method  of  the 
interview  was  reviewed  with  the  interviewee,  and  a  relaxed,  non¬ 
threatening  atmosphere  was  encouraged.  Also,  bearing  in  mind  the 
potential  biasing  effect  of  the  interviewer  questions  and  responses, 
the  structure  of  the  previously  validated  interview  guide  was 
followed  as  much  as  possible  without  dampening  respondent  enthu¬ 
siasm. 

Data  Collection  Plan 

The  universe  for  this  research  was  all  DOD  personnel  involved 
in  the  acquisition  process  who  are  experienced  in  the  use  of  the  RIW 
contracting  strategy.  Within  this  universe  was  the  smaller  population 
of  Air  Force  RIW  knowledgeable  personnel  that  was  sampled.  The 
population  was  those  who  have  worked  with  the  RIW  guidelines  at 
some  time,  either  selecting  candidates /developing  clauses,  or,  those 
personnel  involved  in  the  current  RIW  review  and  administration  pro¬ 
cess.  This  is  not  to  say  that  personnel  must  have  used  RIW  to  be 
considered  in  this  study.  However,  unless  they  have  at  least  con¬ 
sidered  using  the  RIW  technique  in  a  contract,  they  were  not  included 
in  the  population. 

The  sample  was  a  convenience — judgemental  sample  that  was 
based  upon  two  sources.  The  first  source  was  a  semi-annual  RIW 
report  (from  1977  to  date)  prepared  at  both  AFSC  and  AFLC  at  the 
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headquarters  level.  In  this  report  all  RIW  applications  currently 
under  contract,  and  all  contemplated  RIW  applications  are  listed  and 
a  contact  point  is  provided  for  each  item.  The  second  source  of 
sample  candidates  was  personnel  involved  in  the  RIW  review  and 
administration  process.  Part  of  this  selection  was  based  upon  office 
position,  e.g,,  AFSC  and  AFLC  both  have  RIW  monitors,  and  AFLC 
has  personnel  designated  directly  with  RIW  analysis.  Selection  was 
also  based  upon  knowledge  and  experience  within  these  areas. 

Finding  RIW  knowledgeable  personnel  was  a  difficult  task  and  was 
based  mainly  upon  the  indications  gained  during  many  personal  inter¬ 
views  conducted  during  the  initial  stages  of  research.  Because  of 
the  relatively  small  number  of  people  available,  an  attempt  was 
made  to  interview  as  many  people  as  could  be  identified  who  met  the 
established  criteria. 

The  fairly  recent  time  frame  was  selected  because  the  study 
deals  with  "perceptions,  "  and  current  perceptions  were  judged  to  be 
easier  to  obtain  and  more  valid.  The  assumption  was  that  sampled 
personnel  provided  their  honest  perceptions  of  the  adequacy  of  the 
guidelines  and  some  explanation  for  their  opinions. 

While  this  may  be  considered  a  restricted  sample,  it  is  felt 
that  it  is  generalizeable  to  future  AF  programs  that  use  the  RIW 
process,  because  it  is  representative  of  the  mix  of  people  who  will 
be  working  the  programs  in  the  future.  The  individuals  interviewed 
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were  from  different  levels  of  responsibility,  civilian  and  military 
personnel,  with  varying  degrees  of  experience. 

The  collected  data  was  tested  by  individual  question.  However, 
the  questions  were  grouped  under  one  of  the  attributes  of  adequacy 
previously  defined.  Classification  questions  at  the  end  of  the  inter¬ 
view  guide  allowed  sample  stratification  for  clarification  of  the 
results. 

Data  Analysis  Plan 

The  analysis  plan  has  two  major  divisions  that  relate  directly 
to  the  research  questions,  which  deal  with  guideline  adequacy  and 
need  for  change.  The  first  division/question  is  concerned  with  guide¬ 
line  adequacy.  In  order  to  answer  the  questions  of  this  division, 
adequacy  was  described  via  two  attributes  or  characteristics.  The 
interview  questions  were  related  to  these  attributes.  The  questions 
were  designed  to  measure  the  perceived  adequacy  of  the  guidelines 
as  regards  specific  issues  in  the  RTW  area.  The  interview  questions 
can  be  grouped  under  one  of  the  two  attributes  of  adequacy  so  as  to 
allow  analysis  in  terms  of  the  more  specific  attribute  rather  than 
the  broader,  overall  term  adequate.  For  example,  one  attribute  of 
adequate  is  completeness.  An  interview  question  that  would  map 
into  this  attribute  would  be,  "Are  the  guidelines  sufficiently  detailed 
regarding  the  types  of  funds  to  be  used  for  contracts  incorporating 
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RIW?  "  If  analysis  of  this  question  indicates  the  guidelines  are  per¬ 
ceived  as  not  sufficiently  detailed  then  the  guidelines  are  not  adequate, 
and  more  specifically  they  are  not  adequate  because  they  are  not 
complete.  The  relationship  of  interview  questions  to  attributes  to 
the  research  question  presents  a  logical  and  interpretable  analysis 
flow,  that  should  make  the  analysis  clearer  and  provide  adequate 
support  for  the  conclusions. 

Each  interview  question  was  analyzed  separately  to  determine 
adequacy  (via  the  attribute  selected).  The  Kolmogorov-Smirnov 
one-sample  test,  a  goodness  of  fit  test,  was  projected  to  be  used  to 
test  the  distribution  of  scaled  responses  for  the  question,  against  a 
criterion  distribution  which  was  a  theoretical  estimation  of  how  the 
scores  should  have  been  distributed  based  upon  the  judgement  of  the 
researchers.  The  sample  distribution  was  tested  against  the  criter¬ 
ion  distribution  at  the  90%  confidence  level.  The  test  was  structured 
as  follows: 


H0  :  sample  distribution  =  criterion  distribution  (adequate) 

Hj  :  sample  distribution  4  criterion  distribution  (inadequate) 

If  the  null  hypothesis  could  be  rejected  with  90%  confidence,  (i.e.,  90 
out  of  100  times  the  difference  between  the  distributions  was  an 
actual  difference,  rather  than  a  difference  due  to  random  fluctuation) 
then  the  difference  was  accepted  as  statistically  discernable  and  the 
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conclusion  of  inadequacy  was  drawn  through  default  (47:47-52). 

This  exercise  was  repeated  for  each  scaled  response  question. 
For  each  question  in  which  the  null  hypothesis  was  rejected  a  con¬ 
clusion  of  inadequacy  was  drawn. 

The  test  described  above  was  performed  by  computer  using  the 
Statistical  Package  for  the  Social  Sciences  (SPSS),  Version  7  (36;  37). 
In  the  SPSS  package  used  to  perform  the  Kolmogorov-Smirnov  one- 
sample  test,  only  three  theoretical  distributions  (uniform,  normal, 
or  poisson)  were  available  for  use  as  the  criterion  distribution. 
However,  neither  of  these  distributions  were  appropriate  for  this 
test.  Therefore,  the  one  sample  test  could  not  be  performed  unless 
it  was  done  by  hand.  In  order  to  use  the  computer  to  perform  the 
tests,  an  adjustment  was  made.  The  difficulty  was  overcome  by 
performing  the  Kolmogorov-Smirnov  two- sample  test  in  which  the 
criterion  distribution  developed  by  the  researchers  was  used  as  one 
of  the  sample  distributions  (47:127-136). 

The  second  division  of  the  plan  was  tied  mainly  to  the  responses 
received  in  the  first  section.  The  second  section  dealt  with  changes 
that  were  required  in  the  guidelines.  If  the  first  section  indicated 
inadequacy  in  terms  of  either  of  the  attributes  of  adequacy,  the 
second  section  provided  direction  on  the  changes  that  needed  to  be 
made  to  correct  inadequacy.  This  section  tied  together  the  results 
of  the  scaled  responses,  hypothesis  tests  and  the  open  ended 
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responses  in  order  to  develop  recommended  changes.  These  concepts 
can  be  graphically  portrayed  as  indicated  in  Figure  3. 

Interpretation  of  experimental  results  required  more  detailed 
analysis  of  the  data  collected.  This  was  because  the  results  tended 
to  deviate  from  those  hypothesized  by  the  researchers.  The  classifi¬ 
cation  data  collected  was  used  to  stratify  the  sample  and  test  for 
significant  differences  between  subgroups. 

Assumptions 

1.  The  results  of  the  survey  of  the  current  RIW  knowledgeable 
individuals  can  be  generalized  to  future  populations  because  personnel 
policies  and  practices  will  remain  the  same. 

2.  The  interviewees  provided  their  honest  perceptions  of  the  adequacy 
of  the  guidelines  and  changes  for  improvement. 

3.  The  five  point  scale  used  on  the  interview  guide  provided  enough 
catagories  of  choice  so  as  not  to  mask  differences  in  perceptions, 
and  to  make  comparison  meaningless. 

Limitations 

The  hypothesis  test  involved  the  establishment  of  a  theoretical 


distribution  that  reflected  adequacy  and  was  based  on  the  judgement 
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of  the  researchers.  If  this  distribution  was  changed  the  test  results 
could  be  different,  with  correspondingly  different  results. 


ANALYSIS  METHODOLOGY 


CHAPTER  III 


ANALYSIS  OF  RESULTS 


Overview 

This  chapter  presents  the  findings  that  resulted  from  the 
analysis  of  data  collected  via  interview.  The  framework  for  analysis 
was  presented  in  the  previous  chapter  and  was  structured  around  a 
question-by-question  test  of  adequacy  through  scaled  response 
questions.  Other,  fipen  ended,  questions  provided  suggestions  for 
improvement  to  the  guidelines  and  were  a  necessary  elaboration  of 
comments  received  through  the  more  structured  scaled  response 
questions.  Therefore,  in  this  chapter,  the  analysis  results  are 
presented  on  a  question-by-question  basis  for  the  ten  scaled  response 
questions.  The  actual  distribution  of  answers,  and  the  criterion 
distribution  of  answers  for  each  question  are  displayed,  and  the 
results  of  the  hypothesis  test  (given  in  terms  of  a  prob-value,  for 
ease  of  understanding)  are  discussed.  Sensitivity  of  results  based 
on  the  criterion  distribution  are  discussed,  and  population  character¬ 
istics  are  analyzed.  Following  this  discussion  the  results  of  the 
open  ended  questions  are  presented  on  a  similar  question-by-question 
basis. 
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Scaled  Response  Question  Results 

Criterion  distribution.  The  criterion  distribution  was  developed 
based  upon  a  review  of  current  literature  dealing  with  the  actual  and 
potential  problems  and  benefits  of  using  the  trial  concept  of  RIW. 

RIW  guidelines  can  be  a  means  toward  avoiding  problems  with  the 
RIW  technique,  and  a  means  of  encouraging  its  effective  and  efficient 
use.  If  inadequate,  the  guidelines  could  be  a  hindrance,  contributing 
to  misapplication  of  the  technique.  Such  an  occurrence  could  easily 
invalidate  what  may  be  a  valid  RIW  application.  Therefore,  the 
distribution  was  designed  to  be  a  standard  not  only  for  the  guidance 
that  is  currently  needed,  but  also  for  what  is  needed  to  provide  the 
most  complete  and  helpful  guidance  available.  With  this  in  mind  the 
following  distribution  (illustrated  in  terms  of  the  scale  used  on  the 
interview  guide)  was  developed: 

25%  60%  10%  5%  0% 

1  2  3  4  5 

I - 1 - 1 - 1 - 1 

Strongly  Agree  Neither  Disagree  Strongly 

Agree  Agrtee  or  Disagree 

Disagree 

The  conceptual  foundation  underlying  this  distribution  was  discussed 
with  experts  in  the  area  of  RIW  throughout  the  research  and  specifi¬ 
cally  during  the  early  stages  of  data  collection  (7;  31;  48).  In  the 
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opinion  of  the  researchers  this  distribntion  represents  an  optimistic 

r 

bat  certainly  not  unrealistic  estimate  of  how  a  knowledgeable  popula¬ 
tion  would  rate  an  adequate  document  (it  is  important  to  stress  that 
this  distribution  was  built  around  the  assumption  of  a  completely 
adequate  document). 

The  criterion  distribution  was  also  discussed  with  Lt.  Col. 
Goldstein,  HQ  USAF/RDCS,  OPR  for  the  guidelines  (18).  He  pre- 
faced  his  comments  by  stating  any  distribution  he  would  estimate 
could  only  be  based  on  his  personal  intuitive  feel  for  the  area.  He 
felt  his  comments  would  be  as  much  a  guess  as  anything  else,  but 
doubted  that  anyone  could  supply  a  distribution  that  could  be  accepted 
as  authoritative.  He  described  the  following  as  his  estimate  of  the 
criterion  distribution. 

15%  45%  25%  13%  2% 

1  2  3  4  5 

I - 1 - 1 - 1 - 1 

Strongly  Agree  Neither  Disagree  Strongly 

Agree  Agree  or  Disagree 

Disagree 

He  felt  the  distribution  would  be  relatively  low  (15%)  in  the  "Strongly 
Agree"  area,  because  he  felt  few  people  would  be  willing  to  strongly 
agree  that  a  government  document  was  completely  adequate  even  if  it 
was.  In  other  words,  assuming  the  guidelines  actually  were 


completely  adequate  and  a  population  of  RIW  knowledgeable  people 
were  asked  to  rate  the  guideline's  adequacy,  only  15%  would  commit 
themselves  to  a  rating  of  "Strongly  Agree."  He  felt  70%  of  the  popula¬ 
tion  would  rate  within  the  "Agree,  "  or  "Neither  Agree  or  Disagree" 
catagories  with  a  split  out  of  about  45%  and  25%,  respectively,  for  a 
really  well  written,  adequate  document.  The  remaining  15%  would 
rate  (breakout  not  expressed)  within  the  "Disagree"  or  "Strongly 
Disagree"  catagories  because  of  disagreement  with  the  policies  ex¬ 
pressed  in  the  document  (ignoring  adequacy),  or  because  of  other 
outlier  characteristics.  In  the  opinion  of  the  researchers  this  distri¬ 
bution  represented  another  realistic,  but  certainly  less  optimistic, 
view  of  what  the  criterion  distribution  might  look  like  for  adequate 
RIW  guidelines. 

One  final  criterion  distribution  was  developed  in  order  to  test 
the  sensitivity  of  the  test  results.  This  distribution  represented  what 
the  researchers  saw  as  a  pessimistic  view  of  what  the  criterion 
distribution  would  look  like  assuming  the  guidelines  were  completely 
adequate: 


10%  35%  35%  15%  5% 


1  2  3  4  5 


Strongly  Agree  Neither  Disagree  Strongly 

Agree  Agree  or  Disagree 

Disagree 


37 


In  research  of  this  type  it  could  be  argued  that  if  any  improve¬ 
ment  could  be  made  to  existing  guidelines,  some  forum  should  be 
available  to  present  these  ideas.  To  require  a  firm  judgement  of 
"inadequate"  towards  a  document  before  attempting  to  improve  that 
document  would  not  seem  to  be  a  wise  position.  By  accepting  a  more 
optimistic  criterion  distribution  any  potential  problem  areas  should 
be  identified  earlier,  and  allow  earlier  correction.  Going  one  step 
farther,  the  crucial  issue  is  not  the  labeling  of  an  area  adequate, 
marginally  adequate,  decreasing  in  adequacy  or  even  inadequate. 

The  issue  is  finding  ways  of  improving  what  is  currently  in  use. 
Selection  of  an  optimistic  criterion  distribution  would  seem  to  have 
some  advantages.  However,  selection  of  an  unrealistically  optimistic 
criterion  distribution  would  only  distort  the  real  situation.  To  pre¬ 
vent  researcher  bias,  the  results,  based  on  three  separate  tests, 
each  using  a  different  criterion  distribution,  are  presented. 

Actual  distribution  test  results.  The  results  of  the  scaled  response 
portion  of  the  interviews  are  presented  in  a  uniform  tabular  format. 
First,  the  criterion  distributions  are  shown,  followed  by  the  actual 
distributions,  then  the  prob-values  and  finally  the  results  of  the  tests. 
The  two  distributions,  actual  and  criterion,  were  tested  via  the 
Kolmogorov-Smirnov  two-sample  test  to  determine  if  the  distributions 
were  significantly  different.  The  computed  prob-value  indicates  the 
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likelihood  of  these  two  distributions  being  different  due  to  random 
fluctuation.  Therefore,  a  very  low  prob-value  indicates  that  the 
difference  between  the  two  distributions  that  could  be  attributed  to 
random  fluctuation  is  so  unlikely  that  the  difference  can  be  accepted 
as  an  actual  difference.  Because  the  probability  is  so  low  the  null 
hypothesis  (which  is  the  assumption  that  the  distributions  are  the 
same  or,  in  other  words,  that  the  guidelines  are  adequate)  can  be 
rejected.  At  this  point,  the  alternate  hypothesis  (that  the  distribu¬ 
tions  are  not  the  same,  indicating  guideline  inadequacy)  is  accepted 
by  default.  A  tabular  format  was  selected,  because  it  allowed  pre¬ 
sentation  of  detail  without  sacrificing  a  "big  picture"  view  of  the 
main  aspects  of  guideline  adequacy  under  review.  The  results  are 
presented  beginning  with  the  test  using  the  most  optimistic  criterion 
distribution  (called  criterion  1),  then  the  less  optimistic  criterion 
distribution  (called  criterion  2),  and  finally  the  pessimistic  criterion 
distribution  (called  criterion  3).  Refer  to  Table  2  for  a  summary  of 
the  results. 

Comments  on  test  results  based  on  different  criterion  distributions . 
Based  upon  the  first  criterion  distribution,  test  results  indicated 
rejection  of  the  null  hypothesis,  which  assumes  adequacy,  for  all 
questions  except  questions  six  and  eight  (guideline  organization  and 
guideline  readability/ understandability).  By  dropping  to  the  second 
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criterion  distribution,  the  null  hypotheses  for  four  additional  questions 
could  not  be  rejected  (question  three- -funding,  question  four--costs 
and  benefits,  question  seven- -logical  flow  of  guidelines  and  question 
ten — framework  for  cost-benefit  analysis).  Under  the  third  criterion 
distribution,  the  null  hypotheses  for  two  more  questions  could  not  be 
rejected  (question  one- -evaluation  of  candidates  and  question  nine-- 
examples  in  the  guidelines).  Alternately  stated,  tests  with  criterion 
one  indicated  eight  areas  of  inadequacy,  criterion  two  indicated  four 
areas  of  inadequacy,  and  criterion  three  indicated  two  areas  of 
inadequacy.  This  type  of  analysis  has  shown  the  sensitivity  of  the 
results  based  upon  the  different  standards  of  comparison  (the  criterion 
distributions).  In  any  case,  it  also  allows  one  to  assess  the  predom¬ 
inant  problem  areas  in  the  guidelines.  As  shown  in  Table  3,  the 
areas  of  Provisions  and  Administration  were  found  inadequate  under 
all  three  criterion  distributions.  Evaluation  of  Candidates  and 
Examples  were  found  inadequate  under  criterions  1  and  2,  while  four 
other  areas.  Funding,  Identification  of  Costs  and  Benefits,  Logical 
Flow  of  Information,  and  Framework  for  Cost-Benefit  Analysis  were 
found  inadequate  only  under  the  most  optimistic  criterion  distribution 
(1). 

Population  segmentation.  The  question  always  arises  in  research  of 
this  type  whether  the  results  vary  depending  upon  the  characteristics 


Question  Inadequate  under  Criterion: 

#  Area 

1 

2 

3 

2.  Provisions 

X 

X 

X 

5.  Administration 

X 

X 

X 

1.  Evaluation  of  Candidates 

X 

X 

9 .  Examples 

X 

X 

3 .  Funding 

X 

4.  Identification  of 

Costs  and  Benefits 

X 

7.  Logical  Flow  of 

Information 

X 

10.  Framework  for  Cost- 
Benefit  Analysis 

X 

6 .  Organization 

8.  Reading  and  Understanding 

PREDOMINANT  PROBLEM  AREAS 
ACCORDING  TO  DEGREE  OF  INADEQUACY 

Table  3 
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of  the  population.  Three  characteristics  were  chosen  for  investiga¬ 
tion,  a  priori,  based  upon  the  researchers'  perceptions  of  logical 
cause-effect  relationships.  The  first  characteristic  was  command. 
Was  there  a  difference  between  the  AFLC  and  AFSC  responses?  The 
second  characteristic  was  the  number  of  RIW  programs  the  inter¬ 
viewee  had  worked  with.  Would  this  characteristic  make  a  difference 
in  their  responses?  The  final  characteristic  was  the  amount  of  time 
the  interviewee  had  been  involved  with  RIW  programs.  Again,  would 
this  characteristic  make  a  difference  in  the  responses? 

In  order  to  form  more  of  an  opinion  on  whether  these  character¬ 
istics  did  make  a  difference,  the  population  was  segmented  according 
to  the  characteristics,  and  individually  tested.  Again,  each  hypothe¬ 
sis  was  tested  using  three  different  criterion  distributions  which 
provided  a  measure  of  test  sensitivity  with  the  subcatagories  of  the 
population  being  explored.  The  results  of  these  tests  are  shown  in 
Table  4. 

To  obtain  an  overview,  it  is  interesting  to  note  the  total  number 
of  significant  results  for  each  subcatagory.  The  test  of  AFLC  per¬ 
sonnel  (ns  13),  indicated  seven  areas  of  significance  (i.e.,  of 
inadequacy).  The  test  of  AFSC  personnel  (n=10),  also  indicated 
seven  areas  of  significance.  A  comparison  of  program  experience 
resulted  in  a  different  outcome.  The  test  of  personnel  with  exper¬ 
ience  in  only  one  or  two  programs  (n=12)  indicated  three  areas  of 


significance,  while  the  test  of  personnel  with  experience  in  three  or 
more  programs  (n=12)  indicated  thirteen  areas  of  significance.  A 
final  comparison  was  based  on  time  experience  with  RIW.  One  cata- 
gory  had  only  those  personnel  with  three  or  less  years  of  experience 
(n=10),  while  the  other  had  only  those  with  more  than  that  amount 
(n=14).  Here  the  difference  was  six  areas  of  significance  versus 
eleven  areas  of  significance. 

More  detailed  comparisons  and  observations  are  possible 
through  a  question  by  question  analysis  of  the  test  results  contained 
in  Table  4.  These  observations  can  help  clarify  any  underlying 
differences  between  the  subcatagories  of  the  population,  differences 
that  may  be  reflected  through  the  criterion  distributions.  Obviously, 
the  degree  to  which  the  guidelines  are  found  adequate  or  inadequate 
depends  upon  the  criterion  distribution  selected.  The  observations 
drawn  from  a  more  detailed  analysis  of  the  test  results  presented 
in  Table  4  should  allow  the  reader  to  form  an  opinion  on  which  cri¬ 
terion  distribution  is  most  appropriate.  The  results,  given  in  terms 
of  prob-values,  are  fairly  self-explanatory.  However,  the  conclu¬ 
sions  drawn  from  the  results,  especially  as  regards  selection  of  a 
criterion  distribution,  could  be  viewed  as  speculative.  The 
researchers  will  comment  upon  the  results  of  the  tests  between  the 
subcatagories  and  provide  their  conclusions  in  Chapter  IV. 

With  this  framework  established,  the  results  of  the  open  ended 
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questions,  which  address  perceived  problem  areas  or  areas  that  can 
be  improved,  are  presented  next. 

Open  Ended  Response  Question  Results 

Increased  emphasis.  The  first  open  ended  question  asked  whether 
the  interviewees  felt  that  any  areas  presently  covered  in  the  RIW 
guidelines  should  receive  greater  emphasis.  Responses  to  this 
question  indicated  that  a  number  of  individuals  had  experienced  pro¬ 
blems  which  they  felt  might  be  avoided,  if  the  pitfalls  are  emphasized. 

Several  individuals  felt  that  the  penalty  clauses  did  not  ade¬ 
quately  penalize  the  contractor  or  provide  relief  to  the  Air  Force  in 
the  event  that  additional  spares  are  needed  because  of  the  contractor's 
failure  to  perform  to  contract  specifications,  e.g.,  the  failure  rate 
exceeds  the  specified  MTBF  (12;  15;  20;  33). 

The  MTBF  guarantee  is  presented  in  the  RIW  guidelines  as  an 
option.  Several  of  the  interviewees  felt  that  a  MTBF  guarantee  is 
essential  and  that  without  it,  RIW  is  simply  a  maintenance  contract 
(11;  54). 

The  problem  of  unverified  failures  or  retest  OK  (RTOK)  is  a 
recurring  one,  and  some  felt  that  additional  emphasis  is  required 
in  this  area.  This  problem  is  compounded  by  the  fact  that  the  initial 
fault  Isolation  is  accomplished  in  many  cases  using  test  equipment 
supplied  by  the  contractor  (11;  12;  33).  It  was  suggested  that  the 
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RTOK  provision  might  be  implemented  gradually  to  allow  for  maturing 
of  new  equipment  (33). 

Another  potential  problem  area  is  to  adequately-  define  the  terms 
line  replaceable  unit  (LRU),  shop  replaceable  unit  (SRU),  and  module 
in  the  contract,  e.g.,  contract  language  must  clearly  state  that  any 
SRU,  module,  component  not  excluded  in  the  contract  will  be  included 
under  the  RIW  provision  of  the  contract. 

As  originally  conceived,  there  were  no  exclusions  under  the 
failure  free  warranty.  RIW  as  it  has  developed  in  the  Air  Force  has 
a  number  of  exclusions,  but  personnel  interviewed  felt  that  this  is 
another  area  which  should  receive  increased  emphasis  and  that  the 
exclusions  should  be  held  to  an  absolute  minimum,  e.g.,  crash, 
battle  damage,  acts  of  God  (10;  11;  12;  22). 

Although  it  might  be  taken  for  granted  by  many,  several  inter¬ 
viewees  stressed  that  it  is  essential  for  the  Air  Force  to  meet  its 
RIW  obligations  (15;  33). 

Evaluation  of  candidates.  Many  individuals  felt  additional  information 
was  needed  on  application  criteria  used  for  selection  of  likely  candi¬ 
dates  for  RIW.  The  criteria,  as  initially  developed,  were  directed 
toward  Identification  of  items  that  would  conform  to  the  pattern  of 
earlier  successes  achieved  in  the  Navy  under  the  failure-free  war¬ 
ranty  (the  original  RIW).  Early  applications  of  this  concept  were 
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used  in  relatively  stable  environments  as  marketing  techniques  by 
aggressive  manufacturers  who  were  pushing  their  products  under 
the  then  new,  total  or  life  cycle  cost  concept  (48). 

The  Air  Force  attempted  to  find  candidates  in  existing  systems 
for  RIW  at  the  ALCs  in  an  effort  to  kick  off  their  own  program.  After 
extensive  review,  the  AFLC  team  tasked  with  identifying  candidates 
failed  to  come  up  with  even  a  single  candidate.  This  occurred  because 
the  criteria  proved  too  restrictive  as  a  group.  At  this  time,  unwilling 
to  abandon  the  concept,  the  emphasis  was  switched  from  use  of  RIW 
on  existing  systems  to  use  with  developing  systems.  However, 
existing  criteria  remained  virtually  "as  is"  in  the  guidelines  with  the 
caveat  that  all  criteria  need  not  be  met  for  the  technique  to  be  used. 

It  is  at  this  juncture  that  the  need  for  more  information  is  felt. 
Several  felt  there  was  no  guidance  on  ranking  the  criteria  to  discern 
their  relative  importance  (7;  29;  40).  That  is,  if  all  criteria  did  not 
need  to  be  met  then  which  were  the  most  important,  and  did  this 
ranking  change  with  different  programs  and  circumstances?  There 
was  a  range  of  opinion  on  how  structured  the  guidance  in  this  area 
should  be.  It  was  suggested  that  the  guidelines  should  start  trying  to 
achieve  a  sophistication  that  would  allow  an  ultimate  model  (24). 

This  model  would  give  a  positive  or  negative  response  at  the  end  of 
the  process  and  would  extend  to  all  warranty  concepts,  not  just  RIW. 

Of  those  who  commented,  all  felt  a  framework  should  be  developed 
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to  show  how  to  go  about  setting  up  a  ranking  or  algorithmic  approach 
to  candidate  selection.  Various  approaches  were  suggested  including 
a  matrix,  checklist  and  graphical  technique  (7;  12;  24). 

Competition  was  seen  by  many  as  the  key  to  negotiating  a 
successful  RIW  contract,  i.e.,  any  negotiations  after  the  source 
selection  were  certain  to  reduce  the  Air  Force's  potential  benefits 
under  the  contract  (10;  11;  15;  35). 

Another  suggestion  indicated  that  there  should  be  a  methodology 
outlined  for  selection  of  the  subsystems  that  should  be  considered  as 
potential  RIW  candidates.  For  example,  in  a  complex  system  like 
the  F-16,  there  were  approximately  300  subsystems  that  had  to  be 
considered  as  potential  RIW  candidates  (12).  Some  type  of  screening 
technique  was  needed.  One  suggested  method  would  use  a  logistics 
support  cost  model  in  conjunction  with  a  sensitivity  analysis.  The 
Logistic  Support  Cost  model  could  identify  the  cost  drivers,  i.e., 
items  likely  to  cost  the  government  a  large  amount  of  money  over  the 
systems  life  cycle.  The  costs  identified  in  this  way  could  be  weighed 
against  the  costs  of  initiating  and  administering  a  RIW  for  that  sub¬ 
system.  In  conjunction  with  the  logistic  support  cost  model  use,  the 
sensitivity  analysis  would  be  used. 

With  the  suggestion  for  a  framework  also  came  the  suggestion 
for  more  rationale  behind  the  criteria,  and  what  some  of  the  effects 
of  proceeding  without  meeting  a  criteria  may  be  (7;  9;  11;  32;  35;  40). 
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This  would  allow  the  user  to  foresee  some  of  the  problems  that  may 
arise  (or  have  already  arisen  from  past  experience)  and  to  evaluate 
the  importance  for  the  item  under  evaluation.  Because  rationale  and 
explanations  were  lacking  the  terms  were  too  vague.  For  example, 
with  the  term  "moderate  to  high  initial  support  costs"- -what  is  mod¬ 
erate  to  high?  Or,  what  does  the  term,  "cooperative  attitude"  on  the 
part  of  contractors  really  mean?  Further  explanation  for  the  criteria 
would  help  remove  the  vagueness  (40). 

It  was  also  felt  an  additional  criteria  should  be  added.  This 
criteria  would  address  the  importance  of  evaluating  the  relationship 
between  the  prime  and  subcontractor  (10;  29).  This  criteria  would 
become  important  when  the  prime  had  the  contract  and  the  subcon¬ 
tractor  was  doing  the  work. 

Some  felt  the  criteria  that  called  for  a  3  to  5  year  period  for 
the  RIW  was  much  too  short  (15;  20;  53).  They  felt  a  longer  time 
period,  e.g.,  would  provide  a  more  realistic  incentive  for  the  con¬ 
tractor  to  "grow"  the  reliability  of  the  item. 

Provisions.  Almost  all  of  the  interviewees  felt  the  guidance  con- 
cerning  the  provisions  that  should  be  considered  for  inclusion  in  RIW 
clauses  needed  change.  The  main  comment  received  throughout  was 
that  the  lessons  learned,  and  policy  changes  since  the  guidelines  were 
first  published  in  1974,  needed  to  be  incorporated.  One  individual 


felt  the  guidelines  as  a  whole  were  of  little  value  as  presently  written 
apart  from  historical  purposes  (48).  These  comments  extended  to 
the  treatment  of  the  provisions  in  the  guidelines.  It  was  felt  that  not 
only  had  the  details  of  construction  of  the  provisions  changed,  but 
also  the  whole  concept  of  RIW  had  undergone  an  evolution.  Other 
individuals  agreed  with  the  need  to  incorporate  lessons  learned  and 
to  fill  in  blank  spots  (7;  24;  29;  30;  40).  Some  felt  much  greater  depth 
and  detail  were  needed  in  the  guidelines  as  a  whole,  with  emphasis 
upon  the  provisions  (29;  40).  It  was  suggested  that  each  provision 
needed  to  be  explained  in  painstaking  detail  (40).  A  format  was  Sug¬ 
gested  in  which  a  sample  provision  could  be  listed  on  one  page  while 
on  the  next  facing  page  would  be  a  line-by-line  (word-by-word  when 
necessary)  explanation  of  the  rationale  behind  the  wording  of  the 
clause  with  comments  on  the  effects  of  inclusion  or  ommission  of 
those  key  thoughts.  Several  individuals  felt  a  single  master  or  boiler 
plate  clause  could  be  written  to  incorporate  the  essential  elements 
that  needed  to  be  considered  (29:  40;  54).  This  single  clause  would  be 
the  only  example  needed.  It  was  felt  that  having  more  than  one  exam¬ 
ple  was  only  confusing  to  guideline  users,  because  no  rationale  was 
provided  for  the  differences.  Others  agreed  that  additional  rationale 
behind  the  provisions  needed  to  be  supplied  (32;  35). 


Funding.  Several  individuals  felt  funding  guidance  in  the  guidelines 
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needed  change  (3;  18;  24;  29;  40;  48).  All  felt  the  guidance  that  had 
supplemented  the  initial  guidance  (policy  letters  and  memorandum), 
needed  to  be  included  in  the  guidelines,  i.e.,  it  needed  to  be  made 
current.  One  individual  felt  the  guidance  was  inadequate,  because 
it  needed  to  be  in  a  regulation  (48).  That  is,  by  including  this  infor¬ 
mation,  which  is  directive  in  nature,  along  with  the  other  information 
which  is  currently  non-directive  a  potential  for  confusion  is  created. 
Others  felt  that  the  guidance  as  currently  stated  was  correct  as 
regards  fund  type,  but  that  it  needed  to  be  greatly  expanded  to  include 
some  of  the  troublesome  issues  that  surround  this  area  (15;  18;  29; 

38;  40;  46).  For  example,  theiproblems  of  budgeting  before  the  deci¬ 
sion  to  include  a  RIW  as  a  part  of  the  contract  and  of  budgeting  for 
all  the  considerations  (support  equipment,  technical  data,  spare  parts, 
etc.)  of  dropping  the  RIW  and  moving  to  organic  support  were  very 
real  to  the  personnel  involved  with  RIW. 

Costs  and  benefits.  The  main  comment  in  the  area  of  costs  and  bene¬ 
fits  was  that  more  information  was  needed.  The  greatest  need  was 
for  more  cost  information  (7;  9;  20;  29;  30;  32;  35;  40;  48).  It  was 
felt  the  treatment  of  costs  was  not  comprehensive  enough  to  be  really 
helpful.  There  was  no  attempt  to  provide  a  list  or  outline  of  potential 
costs,  especially  some  of  the  intangible  costs  that  may  not  be  obvious 
to  a  user  without  the  experience  gained  from  past  programs.  Also, 
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it  was  felt  an  attempt  should  be  made  to  quantify  or  provide  a  method 
of  quantification  for  some  of  the  intangible  costs  to  allow  the  analysis 
to  be  conducted  more  thoroughly  (48).  Without  a  complete  treatment 
of  the  costs  and  benefits  an  accurate,  trustworthy  analysis  could  not 
be  conducted. 

There  was  a  distinction  drawn  also  between  knowing  what  had 
to  be  considered  and  how  to  actually  go  about  drawing  a  conclusion 
from  the  considerations.  One  individual  felt  some  tools  for  analysis 
ought  to  be  presented  in  the  guidelines  to  at  least  let  the  user  know 
there  are  ways  already  developed  to  analyze  costs  and  benefits  (7). 
One  example  given  was  a  program  developed  for  the  TI  59  pocket 
calculator.  Another  example  given  was  a  recently  completed  analysis 
on  a  program  for  the  B-52  offensive  avionics  system,  which  was 
written  up  and  which,  it  was  felt,  would  provide  an  excellent  frame¬ 
work  for  analysis  on  other  programs. 

A  dm  ini  s  trative .  The  guidelines'  treatment  of  the  administrative  con¬ 
siderations  of  using  RIW  received  the  most  negative  comments  among 
the  interviewees,  of  all  areas  addressed  during  the  research.  One 
individual  stressed  that  the  lack  in  this  area  was  not  an  oversight  of 
the  original  writers,  but  was  a  function  of  the  limited  knowledge 
available  about  this  aspect  of  RIW  (48).  The  information  that  was 
available  at  that  time  was  included.  He  felt  the  area  was  totally 
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inadequate  now  and  that  the  knowledge  gained  since  the  original 
publishing  date  should  be  included  In  the  guidelines.  These  comments 
were  characteristic  of  many  others  interviewed  (7;  12;  16;  18;  20;  21; 
24;  29;  32;  35;  38;  40).  Currently,  there  is  a  large  gap  in  this  area. 
What  is  presented  is  very  simplistic  and  does  not  respond  to  the  needs 
of  the  users.  What  is  needed  is  a  comprehensive  update,  with  all  the 
information  gained  in  the  course  of  the  past  5  years. 

Logical  flow.  The  problem  of  logical  flow,  although  not  addressed 
directly  was  addressed  in  terms  of  the  format  for  presenting  the 
information  in  the  guidelines.  One  suggestion  in  this  area  was  a 
reorganization  of  the  guidelines  around  two  main  themes  (24).  The 
first  theme  was  that  the  information  presented  in  the  guidelines 
should  be  tied  to  the  phases  of  the  acquisition  life  cycle.  It  was  felt 
that  the  guidance  would  need  to  change  depending  upon  the  phase  of 
the  life  cycle  in  which  the  RIW  was  introduced.  For  example,  the 
guidance  needed  at  the  beginning  of  the  full  scale  engineering  develop¬ 
ment  phase  would  be  different  from  that  needed  during  the  demonstra¬ 
tion  and  validation  phase,  or  later  in  the  production  phase.  This 
comment  emphasized  the  amount  of  risk  that  would  come  into  play 
and  the  amount  of  competition  available.  Both  of  these  factors  would 
change  as  the  life  cycle  unfolded,  and  could  play  a  very  important 
part  in  determining  whether  a  RIW  would  be  appropriate,  and  if 
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appropriate,  the  specific  structure  of  the  RIW  clauses.  The  second 
theme  was  that  the  information  should  be  broken  down  by  standard 
contract  format.  For  example,  section  J  of  a  contract  always  deals 
with  special  provisions  under  standard  contract  format.  The  guide¬ 
lines,  under  this  suggestion,  should  have  a  section  entitled  special 
provisions,  in  which  all  guidance  pertaining  to  those  provisions  would 
be  contained. 

Another  suggestion  dealing  with  the  format  for  the  guidelines 
recommended  three  different  volumes,  which  would  form  the  complete 
guidelines  when  combined  (54).  Volume  one  would  be  an  executive 
summary  document  which  would  provide  an  overview  of  RIW,  policy 
guidance  and  general  information.  This  would  be  similar  to  the  pre¬ 
sent  guidelines  only  without  the  examples.  Volume  two  would  be  much 
more  detailed  and  would  ejqplain  how  to  implement  and  administer  a 
RIW  program.  This  section  would  explain  how  to  decide  on  the  level 
to  which  the  RIW  should  be  applied.  It  would  explain  the  details  of 
cost-benefit  analysis--it  would,  in  short,  tell  the  how  and  why  of  RIW. 
Volume  three  would  be  a  volume  of  case  studies  of  actual  past 
programs  that  would  provide  some  of  the  pertinent  detail  of  those 
programs. 

Organization.  Another  suggestion  indicated  a  section  in  which  fre¬ 
quently  asked  questions  were  posed  and  answered  would  be  helpful  (24). 
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This  section  would  give  some  rationale  on  why-  the  question  was  asked, 
i.e.,  the  implications  of  the  question  that  may  not  be  immediately 
obvious,  or  the  situation  from  which  the  question  generally  (or 
specifically)  has  arisen. 

It  was  also  recommended  that  the  guidelines  needed  to  be 
extensively  cross  referenced  with  other  documents  (24).  This  would 
entail  more  than  just  a  cite  in  a  bibliography,  but  would  also  include 
comments  on  the  background,  scope  and  usefulness  of  the  reference 
and  how  it  could  be  used.  The  emphasis  would  be  on  providing  a 
rationale  behitul  the  information  presented  so  changes,  additions  or 
deletions  could  be  made  more  confidently  (i.e.,  with  an  awareness  of 
the  ramifications). 

Two  individuals  felt  the  organization  of  the  guidelines  would  be 
enhanced  by  including  many  more  "how-to"  type  examples  (32;  35). 

One  individual  felt  the  organization  was  lacking  because  the 
treatment  of  areas  as  a  whole  was  too  general  (40).  The  organization 
could  lead  to  the  impression  that  the  subject  of  RIW  only  needed  to  be 
approached  in  a  general  way  without  much  attention  paid  to  the  details. 
Detailed  rationale  was  seen  as  a  necessary  part  of  good  organization. 

Reading;  and  understanding.  The  majority  of  interviewees  felt  they 
had  no  trouble  reading  and  understanding  the  guidelines. 

Examples.  Comments  received  indicated  an  increase  in  detail. 
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currency  and  number  of  examples  was  needed  (32;  35) 


Framework,  The  interviewees,  as  a  whole,  tended  to  disagree  that 
the  guidelines  provided  a  framework  for  analysis  of  costs  and  bene¬ 
fits.  Comments  were  put  forth  to  improve  this  area  in  the  guidelines 
in  conjunction  with  question  four,  that  dealt  with  identification  of 
costs  and  benefits. 

Other.  Many  individuals  suggested  the  content  of  the  guidelines 
should  be  formalized  through  inclusion  in  the  Defense  Acquisition 
Regulation  (DAR)  or  in  an  Air  Force  regulation  (7;  18;  24;  29;  48). 
There  was  some  difference  of  opinion  as  to  how  much  of  the  content 
should  be  formalized.  The  range  went  from  abolishment  of  the 
pamphlet  with  all  current  guidance  going  into  the  DAR  (11)  to  inclusion 
of  the  major  policy  areas  in  the  DAR  or  Air  Force  regulation,  while 
retaining  some  of  the  more  detailed  and  "how  to"  information  in  a 
pamphlet  similar  to  the  present  format  (18;  48).  One  reason  for  a 
formalization  of  the  guidance  was  to  provide  the  authority  needed  to 
make  current  guidance  stronger.  Based  upon  lessons  learned  the 
guidance  is  (or  should  be)  the  best  direction  possible,  and  deviation 
from  this  guidance  should  be  well  supported.  Presently  the  guidance 
is  not  current,  and  one  possible  reason  updating  has  not  been  accom¬ 
plished  could  be  the  lack  of  authority  currently  found  in  the  guidelines. 
Another  reason  for  formalizing  the  guidelines  would  be  to  ensure 


their  wide  dia semination.  The  consensus  of  opinion  among  all  inter¬ 
viewees  was  that  the  guidelines  were  not  always  readily  available  and 
in  some  cases  (based  upon  personal  experience)  the  guidelines  had 
not  even  been  heard  of  or  seen  despite  the  fact  that  they  were  published 
in  July  1974.  Formalizing  the  guidance  through  the  DAR  or  through 
Air  Force  regulations  would  help  extend  the  guidance  and  provide  a 
greater  assurance  that  those  who  wanted  to  review  the  guidance  would 
find  it  easily  available. 


CHAPTER  IV 


CONCLUSIONS  AND  RECOMMENDATIONS 

Overview 

This  chapter  draws  together  the  results  discussed  in  Chapter  III 
and  shows  how  the  research  objectives  established  in  Chapter  I  were 
met.  The  research  objectives  were  to  determine  if  the  Air  Force 
RIW  guidelines  are  adequate  and  to  obtain  recommendations  for 
improving  those  guidelines,  if  necessary.  To  meet  those  objectives, 
24  individuals  who  are  knowledgeable  in  the  area  of  RIW  were  inter¬ 
viewed  by  the  researchers.  The  interviews  were  designed  to  answer 
the  questions  of  whether  Air  Force  personnel  involved  in  the  con¬ 
tracting  process  perceive  the  RIW  guidelines  as  adequate  and  what 
changes  should  be  made  to  the  RIW  guidelines . 

A  conclusion  of  inadequacy  for  portions  of  the  guidelines  was 
made  based  upon  hypothesis  test  of  the  scaled  responses.  These 
conclusions  are  prefaced  by  comments  regarding  differences  between 
subcatagories  of  the  population  and  how  these  differences  were 
reflected  in  the  criterion  distributions.  The  importance  of  the 
criterion  distribution  in  determining  the  perceived  adequacy  or 
inadequacy  of  areas  within  the  guidelines  is  stressed.  Recommenda¬ 
tions  received  during  the  open  ended  question  portion  of  the  interviews 
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address  the  inadequacies  of  the  RIW  guidelines  and  constitute  the 
suggested  changes  to  the  guidelines. 

Questions  that  did  not  fail  the  test  represent  areas  that  cannot 
be  judged  inadequate.  However,  in  some  cases  recommendations 
were  received  which  could  improve  these  areas,  notwithstanding  their 
rating  of  adequate.  Under  these  circumstances  the  recommendations 
were  provided  for  consideration.  Concise  recommendations  are 
presented  which  clearly  address  the  main  problems  perceived  by 
those  in  the  field  who  are  using  the  guidelines.  A  final  section  sug¬ 
gests  areas  for  future  research  that  appear  to  offer  some  potential 
for  contribution. 

Criterion  Distribution 

In  Chapter  III,  observations  were  made  concerning  the  sub- 
catagories  of  the  population  and  how  differences  among  the 
subcatagories  could  be  revealed  through  selection  of  the  criterion 
distribution.  For  example,  test  of  a  hypothesis  using  a  selected 
criterion  distribution,  based  upon  the  population  as  a  whole  will  yield 
a  certain  result.  By  segmenting  the  population  according  to  some 
characteristic,  and  by  testing  each  segment  separately,  using  the 
same  criterion  distribution,  the  result  may  be  different.  If  this  is 
the  case,  it  shows  that  a  portion  of  the  population  felt  differently, 
under  the  selected  criterion,  about  the  hypothesis  being  tested,  but 
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their  responses  were  lost  through  averaging  with  the  responses  of 
the  other  segments  of  the  population.  At  this  point,  an  evaluation 
must  be  made  of  the  importance  of  the  results  received  from  the 
segment  that  felt  differently.  If  the  evaluator  wants  the  differences 
to  be  revealed  in  a  test  of  the  whole  population  (because  he  considers 
them  significant  and  does  not  want  them  to  be  lost),  then  a  more  dis¬ 
criminating  criterion  distribution  would  have  to  be  used.  This 
explanation  concerning  the  criterion  distributions  is  stressed, 
because  a  judgement  of  adequacy  or  inadequacy  depends  upon  which 
criterion  distribution  is  selected. 

Referring  to  the  population  segmentation  tests  (see  Table  4)  the 
observations  below  present  the  researchers'  interpretation  of  the 
results. 

Comparison  of  results  obtained  with  AFSC  and  AFLC  indicated 
that  although  there  are  individual  differences,  overall,  there  does 
not  appear  to  be  much  of  a  difference  between  these  groups.  Com- 
parison  of  the  two  catagories  of  program  experience  revealed  a 
noticeable  difference- -those  with  more  program  experience  saw  the 
guidelines  as  much  more  inadequate,  overall,  than  did  those  with 
less  program  experience.  The  final  comparison  was  based  on  time 
experience  with  RIW.  Here  again  there  was  a  difference,  but  the 
difference  was  not  as  pronounced  as  the  comparison  between  levels 
of  program  experience.  This  result  leads  to  the  observation  that 
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program  experience  seems  to  be  a  greater  influence  than  time  exper¬ 
ience  in  developing  more  discriminating  attitudes. 

As  previously  stated,  the  researchers  developed  criterion  one 
to  use  in  testing  the  interview  results.  If  criterion  two  had  been 
used,  four  less  areas  of  the  guidelines  would  have  tested  as  inade¬ 
quate.  If  criterion  three  had  been  used,  two  less  areas  than  in 
criterion  two  would  have  tested  as  inadequate.  Through  segmentation 
of  the  population,  and  subcatagory  testing,  some  possible  explanations 
for  the  differences  between  the  distributions  can  be  advanced.  The 
reader  can  use  this  information  to  aid  in  his  selection  of  the  most 
appropriate  criterion  distribution. 

Question  three,  dealing  with  funding  guidance,  was  judged 
inadequate  per  criterion  one  but  as  adequate  per  criterion  two  and 
three.  It  is  interesting  to  note  that  the  subcatagories  of  "AFSC,  " 
'less  program  experience"  and  'less  time  experience"  did  not  test 
this  question  as  inadequate  while  subcatagories  "AFLC,  "  "more 
program  experience"  and  "more  time  experience"  did  test  the  area 
as  inadequate  (see  Table  4).  The  difference  appears  to  be  explained 
by  lack  of  experience  and  being  removed  from  the  problem.  AFSC 
is  removed  from  the  funding  problems  of  obtaining  spares,  support 
equipment,  technical  data  and  of  moving  to  organic  support.  This 
lack  of  hands  on  experience,  along  with  general  experience  level 
could  account  for  the  difference  in  the  area. 


Question  four,  dealing  with  identification  of  costs  and  benefits, 
was  similarly  rejected  as  adequate  with  criterion  one  but  not  rejected 
with  criterion  two  and  three.  Again,  it  is  interesting  to  note  that  the 
subcatagories  of  'less  program  experience"  and  "AFSC"  failed  to 
test  this  area  as  inadequate,  while  the  other  subcatagories  either  did 
test  the  area  as  inadequate  or  were  very  close  to  testing  the  area  as 
inadequate.  A  possible  explanation  could  be  that  AFSC  personnel  are 
not  really  aware  of  all  the  downstream  costs  to  the  same  degree  as 
AFLC,  because  they  are  removed  from  the  situation. 

Question  seven,  dealing  with  the  logical  flow  of  the  guidelines, 
was  tested  as  inadequate  via  criterion  one;  adequate  via  criterion  two 
and  three.  The  subcatagories  of  "AFLC"  and  "less  program  exper¬ 
ience"  failed  to  test  this  area  as  inadequate,  while  the  others  tested 
it  inadequate.  Experience,  in  terms  of  programs  again  is  seen  to  be 
an  important  discriminator.  Why  this  difference  did  not  show  up 
under  "less  time  experience"  is  unknown.  Why  "AFLC"  tested  dif¬ 
ferently  than  the  other  catagories  is  subject  to  speculation.  However, 
one  explanation  could  be  that  AFLC,  not  normally  involved  in  the 
selection  and  evaluation  of  candidates,  has  less  need  for  a  document 
that  logically  flows  through  the  various  areas  of  consideration.  That 
is  especially  true,  since  the  guidelines  were  written  mainly  for 
application  during  the  early  stages  of  a  program  (AFSC  being  the 
primary  command  at  that  point) . 
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Question  ten,  dealing  with  the  framework  provided  in  the  guide¬ 
lines  for  cost-benefit  analysis,  was  tested  as  inadequate  via  criterion 
one,  but  adequate  via  criterion  two  and  three.  For  this  test,  exper¬ 
ience  in  terms  of  both  programs  and  time  appeared  to  be  the  explana¬ 
tion  for  the  difference  in  results.  The  other  subcatagories  either 
tested  the  area  as  inadequate  or  were  very  close  to  doing  so  (Table  4) 

Conclusions 

Based  upon  the  analysis  of  results  presented  in  Chapter  III,  the 
guidelines  have  several  areas  of  inadequacy.  This  conclusion  is 
based  on  the  tested  perceptions  of  users  in  the  field,  who  are  know¬ 
ledgeable  of  the  RIW  concept  and  who  have  worked  with  RIW  programs 
Table  5  summarizes  those  areas  (Table  5  is  the  same  as  Table  3). 

Recommendations 

Based  upon  the  interviews,  there  are  areas  in  the  guidelines 
that  need  to  be  especially  emphasized,  because  experience  has  shown 
that  problems  seem  prone  to  arise  in  these  areas.  As  mentioned  in 
Chapter  III,  the  areas  of  penalty  clauses,  MTBF  guarantees,  RTOKs, 
failure  definition,  level  definition,  exclusions,  government  obligations 
and  need  for  competition  all  need  to  be  addressed.  These  problems 
are  characteristic  of  the  many  problems  that  will  occur,  in  various 
forms,  from  program  to  program.  The  researchers  feel  the  best  way 
to  emphasize  an  area  is  to  provide  a  large  measure  of  the  rationale 
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Question 


Inadequate  under  Criterion: 


#  Area 

1 

2 

3 

2.  Provisions 

X 

X 

X 

5 .  Administration 

X 

X 

X 

1.  Evaluation  of  Candidates 

X 

X 

9.  Examples 

X 

X 

3 .  Funding 

X 

4.  Identification  of 

Costs  and  Benefits 

X 

7.  Logical  Flow  of 

Information 

X 

10.  Framework  for  Cost- 
Benefit  Analysis 

X 

6.  Organization 

8.  Reading  and  Understanding 

SUMMARY  OF  CONCLUSIONS 
Table  5 
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behind  why  that  area  is  important  and  how  various  changes  in  the 
area  will  impact  the  program.  Illustration  of  how  this  recommenda¬ 
tion  can  be  applied  will  address  another  problem  area  brought  out 
through  the  interviews — that  of  the  examples  and  contract  provisions 
currently  in  the  guidelines.  The  suggestion  is  to  develop  one  com¬ 
prehensive  example  clause  which  contains  all  the  appropriate  contract 
provisions.  The  provisions  now  in  the  guidelines  are  out  of  date  and 
consequently  have  blank  spots  and  misleading  sections.  Before 
emphasis  can  be  meaningfully  applied  to  the  provisions,  they  should 
be  brought  up  to  date.  A  suggested  format  would  have  the  clause 
listed  down  one  side  of  the  page  with  the  rationale  on  a  facing  page. 
The  clause  would  be  explained  line  by  line,  ot  phrase  by  phrase,  in 
great  detail.  Only  one  example  clause  is  suggested,  because  more 
than  one  example  clause  may  lead  to  confusion.  Detailed  rationale 
would  allow  modification  of  the  provisions  of  the  one  example,  as 
needed,  and  would  provide  the  emphasis  required  to  avoid  the  pitfalls 
experienced  in  the  past. 

Another  major  problem  area  centered  around  the  identification 
of  costs,  and  development  of  an  analysis  framework  to  weigh  the 
economic  costs  and  benefits  of  a  RIW  application.  The  first  suggestion 
is  to  completely  identify  all  costs  by  means  of  a  checklist.  The  idea 
of  a  checklist  may  be  viewed  as  too  simplistic,  but  experience  has 
shown  it  is  very  easy  to  overlook  areas,  until  it  is  too  late  to  take 
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them  into  account.  The  list  should  include  statements  of  the  intangible 
as  well  as  tangible  costs  of  a  RIW.  The  intangible  may  be  extremely 
difficult  to  quantify,  but  the  first  step  toward  quantification  and 
measurement  of  these  costs  is  to  acknowledge  their  existence,  label 
them,  and  start  to  work  with  them.  The  second  suggestion  is  to  pro¬ 
vide  several  examples  of  how  to  analyze  the  costs  and  benefits. 
Detailed  examples  with  rationale  behind  the  components  of  the  analysis 
model  or  technique  should  help  provide  the  proper  emphasis  and 
framework  for  effective  use. 

Selection  of  candidates  is  another  area  that  needs  to  be 
improved.  The  main  suggestion  here  is  to  provide  some  means  of 
ranking  the  criteria,  i.e.,  of  determining  the  relative  importance  of 
the  criteria.  Perhaps  this  could  be  accomplished  through  develop¬ 
ment  of  an  algorithmic/flowchart  approach.  The  other  main  sugges¬ 
tion  should  aid  in  accomplishing  the  first  suggestion.  This  suggestion 
is  to  provide  the  rationale  behind  the  criteria,  with  statements  of 
probable  impact  for  each  criteria  when  it  is  not  met.  Provided  this 
background,  a  user  can  examine  his  own  program  and  more  easily 
determine  the  relative  importance  of  the  criteria. 

The  guidelines'  treatment  of  administrative  functions  and  con¬ 
siderations  is  inadequate.  The  inadequacy  is  due  to  the  fact  that  much 
has  been  learned  in  this  area,  since  the  guidelines  were  published. 
What  was  adequate  then,  no  longer  is  adequate.  The  recommendation 
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here  is  simple- -update  the  section  with  current  policy  and  procedure. 
The  section  in  die  guidelines  should  provide  a  complete  list  of  admin¬ 
istrative  functions  and  considerations,  include  the  rationale  of  why 
the  functions  are  important,  and  explain  some  of  the  possible  pitfalls 
surrounding  the  functions.  In  this  area,  as  in  the  previous  area, 
rationale  is  the  key  thought. 

Funding  guidance  is  smother  area  that  has  been  supplemented 
and  changed  since  publication,  but  the  changes  have  not  been  entered 
into  the  guidelines.  Changes  and  supplements  should  be  incorporated 
into  the  section  to  make  it  current.  This  section  can  be  improved  by 
an  expansion  of  the  material  into  what  could  be  termed,  financial 
management.  Guidance  on  planning  and  budgeting  for  spares,  support 
equipment,  technical  data  and  for  transition  costs  of  moving  from  a 
RIW  to  organic  support  should  be  provided.  Per  the  interviews,  this 
area  has  been  a  very  confusing  and  changing  area  which  should 
receive  more  structure  and  detail. 

The  organization  of  the  guidelines  can  be  improved  through 
addition  of  more  information.  Several  additions  are  suggested  as 
ways  of  making  the  guidelines  more  responsive  to  the  user's  needs. 
An  annotated  bibliography  should  be  provided  as  a  reference  source 
for  users  who  want  a  variety  of  viewpoints  and  a  broader  background 
of  the  RIW  technique  and  its  applications.  A  section  listing  points  of 
contact  for  RIW  programs,  past  and  present,  should  be  included  as 
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a  user  resource.  A  section  on  analysis  models  should  be  presented 
for  the  user's  review.  A  section  with  frequently  asked  questions  and 
appropriate  answers  should  be  provided  to  amplify  important  areas. 

The  logical  flow  of  the  guidelines  can  be  improved  by  changing 
the  format  of  the  guidelines.  The  guidelines  should  consist  of  three 
volumes.  The  first  volume  would  be  statements  of  general  policy  or 
an  executive  summary.  It  would  provide  an  overview  of  RIW,  and 
general  information  and  policy.  Volume  two  would  be  the  detailed 
"how  to"  volume  in  which  information  on  analysis,  implementation 
and  administration  would  be  provided.  Volume  three  would  be  a 
collection  of  case  studies,  additional  information  such  as  the  anno¬ 
tated  bibliography  mentioned  earlier,  and  other  helpful  guidance.  All 
three  volumes,  especially  volumes  one  and  two,  should  stress  the 
relationship  of  the  user's  actions,  and  acquisition  life  cycle,  and 
other  life  cycle  cost  techniques. 

In  order  to  assure  ready  availability  of  current  RIW  guidance, 
and  especially  to  provide  more  authority  to  the  guidance  which  has 
been  accepted  as  valid,  the  guidelines  should  be  formalized  through 
inclusion  into  the  Defense  Acquisition  Regulation  (DAR). 

A  summary  of  the  recommendations  is  contained,  in  abbreviated 
form,  in  Table  6. 
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ELEMENTS  TO  CONSIDER  IN  CHOOSING  CANDIDATES  FOR  RIW 

1.  Rank  criteria  according  to  relative  importance. 

2.  Conetrnct  a  screening  procedure  inch  as  a  checklist,  algorithm  or  matrix. 


PROVISIONS  NORMALLY  INCLUDED  IN  RIW  CLAUSES 

3 .  Develop  a  current  example  that  includes  all  relevant  provisions .  This 
example  should  be  fully  explained,  so  a  user  could  understand  the 
ramifications  of  any  changes  to  the  basic  example. 


TYPES  OF  FUNDS  TO  BE  USED  FOR  RIW  CONTRACTS 

4.  Publish  all  pertinent  policy  guidance  in  appropriate  regulations. 

5.  Develop  a  method  to  Initially  budget  for  RIW  and  its  alternatives. 

6.  Develop  a  timetable  (planning)  to  budget  for  transition  from  RIW  to 
organic  support. 


POTENTIAL  COSTS  AND  BENEFITS  TO  BE  DERIVED  FROM  A  RIW 

7.  Identify  all  relevant  coats  and  benefits  and  incorporate  them  in  a  checklist. 

8.  Develop  a  model  and  "how  to"  examples. 


ADMINISTERING  CONTRACTS  INCORPORATING  RIW 

9.  Identify  all  relevant  RIW  administrative  functions. 

10.  Ensure  all  information  provided  is  current. 


ORGANIZATION  OF  THE  RIW  GUIDELINES 

11.  Develop  an  annotated  bibliography. 

12.  Include  points  of  contact. 

13 .  Compile  a  list  of  frequently  asked  questions  and  answers . 

14.  Incorporate  a  chapter  on  cost  evaluation  models.  i 


LOGICAL  FLOW  THROUGH  ALL  RELEVANT  CONSIDERATIONS 

19.  Tie  the  guidelines  to  the  phases  of  the  acquisition  life  cycle. 

16.  Divide  the  guidelines  into  three  volumes:  basic  policy,  detailed  "how  to" 
information,  and  case  studies. 


EXAMPLES  OF  HOW  TO  WRITE  RIW  CLAUSES 

17.  See  recommendation  3  above. 


OTHER 

18.  Incorporate  RIW  guidance  into  the  Defense  Acquisition  Regulation. 

19.  Change  the  DAR  as  required  to  keep  people  in  the  field  abreast  of  the 
latest  lessons  learned,  procedural  modifications,  and  policy  changes . 


SUMMARY  OF  RECOMMENDATIONS 
Table  6 


Recommendations  for  Further  Research 

The  researchers  believe  the  following  RlW-related  subjects 
offer  significant  potential  for  future  research  efforts: 

1.  Completed  and  active  RIW  contracts  should  be  analyzed  to 
determine  whether  RIW  is  an  effective  contracting  technique.  AFLC 
headquarters  is  currently  the  repository  of  all  RIW  information  pre¬ 
viously  contained  in  the  ARINC  operated  Warranty  Information 
Center.  The  information  available  should  provide  a  useable  data 
base  for  evaluation  of  the  RIW  concept.  Alternatively,  a  methodology 
could  be  developed  for  evaluation  of  the  data  as  a  first  step. 

2.  Researchers  should  explore  development  of  new  models  or 
testing  of  existing  models  to  be  used  in  determining  the  cost  effec¬ 
tiveness  of  a  RIW  candidate. 

3.  A  decision  tree  or  algorithm  to  evaluate  the  applicability  of 
a  RIW  candidate  should  be  developed. 

4.  Evaluation  of  why  contractor  initiated,  no-cost  to  the 
government,  ECPs  have  not  been  used  as  much  as  expected  to  achieve 
greater  reliability  growth  during  the  warranty  period  should  be 
accomplished.  With  this  evaluation  should  be  an  exploration  of  ways 
to  increase  contractor  effort  in  this  area.  For  example,  one  sug¬ 
gestion  received  during  research  was  for  the  Air  Force  to  share  the 
cost  of  the  ECPs  or  to  use  value  engineering  clauses  to  incentivize 
reliability  growth,  if  the  contract  was  more  than  50%  complete  or  if 


the  MTBF  target  had  been  exceeded  (29) .  Along  these  lines,  are 
significant  changes  being  made  which  are  not  within  the  scope  of  the 
ECP  and  configuration  control  programs? 

5.  The  RIW  concept  has  been  a  trial  concept  for  over  five  years 
and  data  is  being  collected  to  evaluate  the  cost  effectiveness  of  the 
technique.  Comments  received  during  research  indicated  data 
requirements  for  RIW  programs  maybe  excessive  (30).  That  is,  the 
government  is  paying  for  data  it  may  not  use.  This  is  because,  some 
speculate,  the  concept  probably  never  really  can  be  evaluated.  The 
research  would  evaluate  present  data  requirements  on  a  cost-benefit 
basis. 

6.  Section  1-324  of  the  DAR,  which  deals  with  warranties, 
should  be  reviewed  to  evaluate  the  section’s  effectiveness  in  providing 
guidance  on  how  to  consider  various  warranties.  Development  of  a 
go-no  go  type  model  is  contemplated.  One  individual  contacted  during 
research  felt  what  to  consider  was  adequately  covered  in  the  DAR  but 
rationale  wasn't  provided  on  how  to  consider  warranties  (24).  It  was 
felt  that  if  a  RIW  was  being  considered,  the  RIW  must  be  considered 
within  the  context  of  warranties  in  general.  Since  the  DAR  is  the 
basic  guidance  used  by  contracting  personnel,  it  was  felt  the  DAR 
must  provide  a  reasoned  path  through  all  the  warranty  concepts  with 
RIW  as  one  of  the  possible  outcomes.  If  RIW  was  the  outcome  of  the 
review,  the  specific  RIW  application  criteria  would  be  reviewed  for  a 


decision  on  whether  or  not  to  use  R1W. 

7.  When  failure  of  an  item  is  due  to  a  cause  which  excludes  it 
from  repair  under  the  terms  of  the  RIW  contract,  its  repair  comes 
under  a  separate  contract  at  additional  cost  to  the  government.  It  is 
suggested  that  the  cost  of  RIW,  with  only  battle  damage  and  Acts  of 
God  excluded,  be  investigated  for  cost  effectiveness  as  compared  to 
the  present  practice  of  having  a  separate  repair  contract  for  a  larger 
number  of  exclusions,  e.g.,  induced  failures,  transportation  damage, 
unauthorized  maintenance,  etc. 

8.  To  date  there  have  been  few  applications  of  RIW  to  mechan¬ 
ical  equipment.  Such  items  represent  a  potentially  beneficial  area 
for  RIW  application.  Future  researchers  should  determine  the  reason 
that  RIW  has  not  been  applied  to  more  mechanical  items.  Addition¬ 
ally,  a  strategy  or  methodology  should  be  developed  to  ensure  that 
mechanical  items  are  included  in  future  evaluations  of  RIW  candidates. 

9.  Future  researchers  should  explore  whether  the  length  of  the 
production  phase  has  any  effect  of  the  effectiveness  of  the  RIW.  It  is 
conceivable  that  a  longer  production  run  would  lead  to  an  increased 
number  of  ECPs,  thereby  improving  reliability. 
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INTERVIEW  GUIDE 


Introduction 

As  was  indicated  in  our  initial  contact  with  you,  we  are  students 
in  the  AFIT  School  of  Systems  and  Logistics.  Our  thesis  work  is  in 
the  area  of  the  Reliability  Improvement  Warranty.  We  are  attempting 
to  assess  the  perceived  adequacy  of  the  RIW  guidelines,  incorporating 
lessons  learned. 

In  answering  the  questions,  we  would  ask  you  to  relate  to  the 

-•  - 

current  guidelines,  i.e.,  the  interim  Air  Force  Reliability  Improve¬ 
ment  Warranty  Guidelines,  referenced  in  the  AFLC  and  AFSC  supple¬ 
ments  to  the  Defense  Acquisition  Regulation. 

There  are  three  types  of  questions.  The  first  ten  questions  are 
designed  to  be  answered  by  indicating  how  much  you  agree  with  a 
statement  concerning  the  RIW  guidelines,  rated  on  a  five  point  scale. 
We  realize  your  answer  will  normally  represent  a  weighted  average 
or  a  mental  balance  of  the  conflicting  factors  within  the  answer.  But, 
in  order  to  make  sure  we  accomplish  this  portion  we  ask  that  you 
reserve  your  comments  concerning  the  rationale  behind  your  answer 
until  the  second  portion  of  the  interview.  In  the  second  portion,  we 
will  ask  seven  open  ended  questions  to  obtain  your  comments  and 
insights.  At  this  time  we  would  welcome  elaboration  on  problem 


areas  brought  to  mind  in  the  first  section.  Finally,  we  will  ask  four 
classification  questions  to  help  us  analyze  the  results  of  our  inter* 
views. 

We  would  like  to  tape  record  the  interview,  to  avoid  the  possi¬ 
bility  of  missing  some  valuable  comment  while  trying  to  take  notes. 
However,  if  you  would  rather  not  be  recorded,  we  will  not  record. 
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The  following  scale  will  be  used  for  questions  1-10: 

5 


Strongly 

Disagree 


COMPLETE 

1.  The  guidelines  contain  sufficient  information  on  the  essential 
elements  to  consider  in  choosing  appropriate  candidates  for  RIW. 

2.  The  guidelines  contain  sufficient  information  concerning  the  provi¬ 
sions  that  should  normally  be  considered  for  inclusion  in  RIW  clauses. 

3.  The  guidelines  are  sufficiently  detailed  to  permit  a  determination 
regarding  the  types  of  funds  to  be  used  for  contracts  incorporating 
RIW. 

4.  The  guidelines  provide  a  sufficient  treatment  of  the  potential  costs 
and  benefits  (both  tangible  and  intangible)  to  be  derived  from  a  RIW. 

5.  The  guidelines  provide  sufficient  information  concerning  the 
establishment  of  efficient  and  effective  procedures  for  administering 
contracts  incorporating  RIW  (including  records  and  reports). 

HELPFUL 

6.  The  guidelines  are  well  organized,  i.e.,  material  on  specific 
issues  is  easily  located. 

7 .  The  guidelines  are  constructed  in  such  a  way  that  they  encourage 
a  logical  flow  through  all  relevant  considerations  in  the  RIW  issues. 

8.  I  have  no  trouble  reading  and  understanding  the  guidelines. 

9.  I  find  the  examples  in  the  guidelines  on  how  to  write  RIW  clauses 
particularly  helpful. 

10.  The  guidelines  provide  a  helpful  framework  for  analyzing  the 
costs  and  benefits  associated  with  a  potential  RIW. 


1 

1 _ 

2 

| 

3 

1 

4 

i 

1 

Strongly 

1 

Agree 

1 

Neither 

\ 

Disagree 

Agree 

Agree  or 

Disagree 
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CHANGES 


1.  What  areas  in  the  guidelines  should  receive  increased  emphasis? 

2.  What  specific  recommendations  might  you  have  for  improving  the 
guidelines  with  respect  to: 

a.  Trade-off  analysis /cost-benefit  analysis? 

b.  Administrative  procedures? 

c.  Funding? 

d.  Construction/organization  of  the  guidelines? 

e.  Examples  provided? 

f .  Other? 

CLASSIFICATION 

1 .  Command 

2.  Number  of  RIW  programs  involved  with 

3.  Length  of  time  involved  with  RIW  type  programs 

4.  Grade  or  rank 
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